- System Dependencies
- Running CI/CD Locally
- Components
- Git Workflow and Versioning
- Auto-generated Build Pipeline
- Step 1 - Identify Components
- Step 2 - Build a Directed Graph by adding a Vertex for every Component and adding an Edges to every Dependency of that Component
- Step 3 - Find Vertices without Outgoing Edges (i.e. no Dependency)
- Step 4 - Create first parallel stage
- Step 5 - Remove Vertices from previous step and find Vertices without Outgoing Edges (i.e. no dependency)
- Step 4 - Create second parallel stage
- Step 5 - Remove Vertices from previous step and find Vertices without Outgoing Edges (i.e. no dependency)
- Step 6 - Create third parallel stage
- Step 7 - Remove Vertices from previous step and find Vertices without Outgoing Edges (i.e. no dependency)
- Step 8 - Create forth parallel stage
- Decisions
- Notes
- N-1 Finding all build-pipeline-generator test cases with a tree graph
- Visual Studio Code
.vscodecontains useful extensions for documentation authoring
- IntelliJ IDEA Community Edition
$ make envfile- Set
GITHUB_PERSONAL_ACCESS_TOKENto update Build Status in Github - Set
NEXUS_ADMIN_PASSWORDto a secure & private password for the Nexus Admin User - Set
NEXUS_ADMIN_USERSto a set of Users whom will have the Admin Role in Nexus, see below for the expected formatting - Set
NEXUS_DEPLOYER_USERSto a set of Users whom will have the Deployer Role in Nexus - Set
SMEE_IDto receive webhooks from Github, provided by smee.io
Similar to HTTP Set-Cookie, key and value are separated by =, key-value pairs are separated by ; and objects (collection of key-value pairs) are separated by :. Be mindful of special characters in passwords.
Basic format:
key=value;key=value:key=value;key=value
Full example:
username=test;firstname=Test;lastname=User;[email protected];password=test123;:username=test2;firstname=Test2;lastname=User;[email protected];password=test123;
Every User should have:
usernamefirstnamelastnameemailpassword
To create the images for CI/CD before deployment run: $ make bootstrap
The CI/CD stack will bind to port 80 and 443
Run the stack: $ docker-compose up
Once the stack is running, to access HTTPS run:
$ step ca bootstrap --ca-url https://ca-127-0-0-1.nip.io --install --fingerprint 84a033e84196f73bd593fad7a63e509e57fd982f02084359c4e8c5c864efc27d`
Jenkins will be available on https://jenkins-127-0-0-1.nip.io
Nexus will be available on https://nexus-127-0-0-1.nip.io
Docker Repository will be available on https://docker-repository-127-0-0-1.nip.io
$ docker-compose down
- MUST have an unique name
.
├── deps List of dependencies
├── Makefile ci target required
└── VERSION Component version (semver)
app1
lib2
ci: deps test build publish
deps:
echo "Installing Dependencies"
test:
echo "Testing..."
build:
echo "Building..."
publish:
echo "Publishing..."
───────────────────o───────────o───────── master
╱ ╲ ╱ ╲
╱ ╲───────o╴╴╴o hotfix/*
╱ ╲ ╱ ╲
o───o╴╴╴╴╴╴╴o ╱ ╲ release/*
╱ ╲ ╱ ╱ ╲ ╱ ╲
╱ o ╱ ╲ ╱ ╲ bugfix/*
╱ ╱ ╲ ╱ ╲
───o───o───o───o───o───────o───────────o─ develop
╲ ╱ ╲ ╱ ╲ ╱ ╲ ╱
o o o o feature/*
All commits into master are Fast-Forward Only --ff-only Merges, CI Release Builds occur just before merging, running this build again is a no-op since the artifacts have already been published. All commits in master will point to the original CI Release Build. release/* and hotfix/* branches target master.
Functionally the same as master and more. First release/* and hotfix/* branches are integrated back into develop, this involves locking develop during this process since its the only point in the workflow where a No Fast-Forward --no-ff merge occurs to keep the history with master intact. feature/* branches target develop
───o───o develop
╲ ╱
o feature/*
VERSION examples:
repo : 6.4.0-dev.8 ⇒ 6.4.0-dev.9
component-a : 3.4.0-dev.0 ⇒ 3.4.0-dev.1
component-b : 1.2.0-dev.5 ⇒ 1.2.0-dev.6
Workflow:
- Async: Checkout new branch
feature/*fromdevelop - Async: Push Commits to
feature/*- Sync: Bump
major/minorversion bump with-dev.0pre-release, if first change to component - Sync: CI Snapshot Build
- Sync: Bump
- Async: Merge
At a point in time, a decision is made to take what has been developed (in develop) and promote to production (in master). release/* allows testing on a 'snapshot' of changes and fix issues with bugfix/* branches.
─────────────────o─────── master
╱ ╲
o───o╴╴╴o release/*
╱ ╲ ╱ ╱ ╲
╱ o ╱ ╲ bugfix/*
╱ ╱ ╲
───o───o───o───o───────o─ develop
VERSION examples (merging to master):
repo : 6.4.0-dev.8 ⇒ 6.4.0
component-a : 3.4.0-dev.0 ⇒ 3.4.0
component-b : 1.2.0-dev.5 ⇒ 1.2.0
Workflow:
- Async: Checkout new branch
release/*fromdevelop- Sync: replace all
-dev.xpre-release version bump with-rc.0 - Sync: CI Build
- Sync: replace all
- Async: Checkout new branch
bugfix/*fromrelease/*, seebugfix/*workflow - Async: 1st Merge
- Sync: drop pre-release version
- Sync: CI Release Build
- Sync:
masterFast-Forward Only--ff-onlyMerge inrelease/*
- Async: 2nd Merge
Functionally the same as feature/* but is created and merged into release/*. Pre-release versions are -rc.x
───o───o╴ release/*
╲ ╱
o bugfix/*
VERSION examples:
repo : 6.4.0-rc.8 ⇒ 6.4.0-rc.9
component-a : 3.4.0-rc.0 ⇒ 3.4.0-rc.1
component-b : 1.2.0-rc.5 ⇒ 1.2.0-rc.6
Workflow:
- Async: Checkout new branch
bugfix/*fromrelease/* - Async: Push Commits to
bugfix/*- Sync:
minorversion bump with-rc.0pre-release, if first change to component - Sync: CI Snapshot Build
- Sync:
- Async: Merge
Functionally the same as release/* but is created from master and contains a single commit. Components are bumped patch. Pre-release versions are -hotfix.
Note if there is a release/* branch, the 2nd merge target (develop) is replaced with that release/* branch.
───o───o───── master
╲ ╱ ╲
o╴╴╴o hotfix/*
╱ ╲
───────o───o─ develop
VERSION examples (merging to master):
repo : 6.4.0 ⇒ 6.4.1
component-a : 3.4.0 ⇒ 3.4.1
component-b : 1.2.0 ⇒ 1.2.1
Workflow:
- Async: Checkout new branch
hotfix/*fromdevelop - Async: Push Commits to
hotfix/*- Sync:
patchversion bump, if first change to component - Sync: CI Snapshot Build
- Sync:
- Async: 1st Merge
- Sync: drop pre-release version
- Sync: CI Release Build
- Sync:
masterFast-Forward Only--ff-onlyMerge inhotfix/*
- Async: 2nd Merge
- Find all
VERSIONfiles, the directory is a Component - Filter out Components that have Git Tags for its
VERSION
Step 2 - Build a Directed Graph by adding a Vertex for every Component and adding an Edges to every Dependency of that Component
.
├── tests1
│ ├── app1
│ │ ├── lib1
│ │ └── lib3
│ │ └── lib2
│ └── app2
│ └── lib3
│ └── lib2
└── tests2
└── app2
└── lib3
└── lib2
Found Vertices labeled with *
.
├── tests1
│ ├── app1
│ │ ├── lib1 *
│ │ └── lib3
│ │ └── lib2 *
│ └── app2
│ └── lib3
│ └── lib2 *
└── tests2
└── app2
└── lib3
└── lib2 *
.
├── lib1
└── lib2
Step 5 - Remove Vertices from previous step and find Vertices without Outgoing Edges (i.e. no dependency)
Found Vertices labeled with *
.
├── tests1
│ ├── app1
│ │ └── lib3 *
│ └── app2
│ └── lib3 *
└── tests2
└── app2
└── lib3 *
.
├── lib1
└── lib2 ───── lib3
Step 5 - Remove Vertices from previous step and find Vertices without Outgoing Edges (i.e. no dependency)
Found Vertices labeled with *
.
├── tests1
│ ├── app1 *
│ └── app2 *
└── tests2
└── app2 *
.
├── lib1 ───────────────┬── app1
└── lib2 ───── lib3 ──┴── app2
Step 7 - Remove Vertices from previous step and find Vertices without Outgoing Edges (i.e. no dependency)
Found Vertices labeled with *
.
├── tests1 *
└── tests2 *
.
├── lib1 ───────────────┬── app1 ──┬── tests1
└── lib2 ───── lib3 ──┴── app2 ──┴── tests2
- Outcome:
- Option 1:
VERSIONfile
- Option 1:
- Options:
- Option 1:
VERSIONfile- Pros:
- Automated in CI, if build triggers omit automated commits
- Searchable in repo history, if part of commit message
- Consistent across components
- Cons:
- Breaks "commit sha" ⇒ "component version" relationship, if automated
- Keeping Git Tag and
package.jsonin sync
- Pros:
- Option 2: Git Tag
- Pros:
- Automated in CI, if build triggers omit tags pushed
- Searchable in repo history, if all tags are pulled
- Git Tagging should be done regardless other Options as best practice
- Strict "commit sha" ⇒ "component version" relationship
- CI Shallow Builds, using Tag as an indicator of a build already done
- Consistent across components
- Cons:
- Keeping Git Tag and
package.jsonin sync
- Keeping Git Tag and
- Pros:
- Option 3:
package.json/build.gradle- Pros:
- Aligns with component framework standards (npm / maven)
- Automated in CI, if build triggers omit automated commits
- Searchable in repo history, if part of commit message
- Cons:
- Inconsistent across components
- Breaks "commit sha" ⇒ "version" relationship, if automated
- Pros:
- Option 1:
- Recommended Option:
- Option 1:
VERSIONfile- Manually triggered via Make:
$ make version-<component>-patch$ make version-<component>-minor$ make version-<component>-major
- Make Targets updates:
- Component
VERSIONandpackage.json - Dependent Components
package.json
- Component
- Build Pipeline creates Git Tags upon pipeline completion
- Manually triggered via Make:
- Option 1:
- Outcome:
- Option 2: Semantic Versioning per component
- Options:
- Option 1: Aligned to Monorepo
VERSION- Pros:
- Consistent across components
- Cons:
- Breaking changes across components are not defined as a major version bump
- Pros:
- Option 2: Semantic Versioning per component
- Pros:
- Existing components migrated to Monorepo can retain versioning lineage
- Cons:
- Each component requires versioning step
- Each dependent dependencies require change
- Pros:
- Option 1: Aligned to Monorepo
- Recommended Option:
- Option 2: Semantic Versioning per component
- Build Pipeline asserts diff on component ⇒ version bump
- Build Pipeline asserts dependent requires change if dependencies change
- This doesn't 100% assert that dependent change actually targets new dependency version, but should be captured during code review
- Could use dependency reports to check pre-build
- This doesn't 100% assert that dependent change actually targets new dependency version, but should be captured during code review
- Option 2: Semantic Versioning per component
- Outcome:
- Option 2: Static Pipeline Generation
- Options:
- Option 1: Dynamic Pipeline Generation
- Pros:
- Always consistent with the structure / changes to the Monorepo
- Can be combined with Shallow Builds to further improve pipeline parallelisation
- Cons:
- Requires custom Jenkins plugin to include the graph library needed to construct the build pipeline on startup
- The resulting build pipeline can't be examined until after the build is complete
- Restricted to Scripted Pipeline, can't include declarative pipeline fragments
- Pros:
- Option 2: Static Pipeline Generation
- Pros:
- Can support Component
Jenkinsfilefragment, the number of stages and what Agent is used for each stage is possible - Produces a standard
Jenkinsfilefile that can be examined to debug issues - Can run in any Jenkins instance without custom plugins or settings
- Declarative Pipeline
- Can support Component
- Cons:
- Requires tooling, validation and git manipulation to keep consistent with the structure / changes to the Monorepo
- Can't produce a pipeline that maximizes pipeline parallelisation using Shallow Builds, the pipeline 'structure' is always the same
- Pros:
- Option 1: Dynamic Pipeline Generation
- Recommended Option:
- Option 2: Static Pipeline Generation
- Alike D-2, the Monorepo already needs to deal with
VERSIONconsistency by validating and updating inline to git commits, - Build Pipeline will generate the pipeline and asserts diff on component ⇒ requires
Jenkinsfileupdate using the build-pipeline-generator tool - Tooling can be enhanced to automatically update the
Jenkinsfileand amend the commit.
- Alike D-2, the Monorepo already needs to deal with
- Option 2: Static Pipeline Generation
- Outcome:
- Option 3: Minimal
Jenkinsfile
- Option 3: Minimal
- Options:
- Option 1:
Makefile- Pros:
- No additional
Jenkinsfile
- No additional
- Cons:
- Can't define the Agent to build on (non 3 musketeers builds)
- Pros:
- Option 2: Complete
Jenkinsfile- Pros:
- Consistent pattern to non Monorepo components
- Cons:
- Requires tooling to extract stages since only 1
pipelinecan be loaded in a build
- Requires tooling to extract stages since only 1
- Pros:
- Option 3: Minimal
Jenkinsfile- Pros:
- Can define the Agent to build on
- Can define multiple stages
- Cons:
- Not a standard structure it starts with
stagenotpipeline
- Not a standard structure it starts with
- Pros:
- Option 1:
- Recommended Option:
- Option 3: Minimal
Jenkinsfile- Given D-3, the build-pipeline-generator combines all
Jenkinsfilefragments into a complete top-levelJenkinsfile - Tooling can be enhanced to deal with no
Jenkinsfileand run acitarget defined in theMakefile
- Given D-3, the build-pipeline-generator combines all
- Option 3: Minimal
Regex
test\(`[\n\s\w└─├│.]+`, \(\) => \{