Skip to content

Commit

Permalink
Merge branch 'master' into hkm/gha-using-devx
Browse files Browse the repository at this point in the history
  • Loading branch information
angerman authored Apr 17, 2024
2 parents cd3556d + 4e1d652 commit d13b6d6
Show file tree
Hide file tree
Showing 99 changed files with 5,318 additions and 2,676 deletions.
12 changes: 12 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"image":"ghcr.io/input-output-hk/devx-devcontainer:x86_64-linux.ghc96-iog",
"onCreateCommand": "on-create-command",
"customizations":{
"vscode":{
"extensions":[
"haskell.haskell"
]
}
// Do not set custom `settings` as they would override devx-container defaults...
}
}
42 changes: 42 additions & 0 deletions .github/workflows/gh-pages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: Github Pages

on:
push:
branches: [ "master" ]
workflow_dispatch:

jobs:
gh-pages:
runs-on: ubuntu-latest
permissions:
contents: write

steps:
- uses: actions/checkout@v3
- name: Install nix
uses: cachix/install-nix-action@v20
with:
nix_path: nixpkgs=channel:nixos-unstable
extra_nix_config: |
experimental-features = nix-command flakes
trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=
substituters = https://cache.iog.io https://cache.nixos.org/ https://cache.zw3rk.com
- name: Build projects and haddocks
run: nix develop --command bash -c "cabal update && cabal build --enable-documentation all && ./scripts/haddocks.sh"
- name: Add files
run: |
git config --local user.name ${{ github.actor }}
git config --local user.email "${{ github.actor }}@users.noreply.github.com"
cp -r ./haddocks/* ./
rm -rf haddocks
git add -A --force
git commit -m "Updated"
- name: Push to gh-pages
uses: ad-m/[email protected]
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
branch: gh-pages
force: true
directory: .

51 changes: 40 additions & 11 deletions .github/workflows/haskell.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ name: Haskell CI
on:
pull_request:
branches: [ "**" ]
push:
branches: [ "master" ]

jobs:
whitespace:
Expand Down Expand Up @@ -34,7 +36,7 @@ jobs:
strategy:
fail-fast: false
matrix:
ghc: ["8.10.7", "9.2.7", "9.6.1"]
ghc: ["8.10.7", "9.2.7", "9.6.4", "9.8.1"]
os: [ubuntu-latest, macos-latest, windows-latest]

env:
Expand Down Expand Up @@ -62,36 +64,53 @@ jobs:
- name: Configure build
shell: bash
run: |
if [ "${{github.event.inputs.tests}}" == "all" ]; then
echo "Reconfigure cabal projects to run tests for all dependencies"
sed -i 's|tests: False|tests: True|g' cabal.project
fi
cp ".github/workflows/cabal.project.local.ci.$(uname -s)" cabal.project.local
echo "# cabal.project.local"
cat cabal.project.local
# A dry run `build all` operation does *NOT* downlaod anything, it just looks at the package
# indices to generate an install plan.
- name: Build dry run
run: cabal build all --enable-tests --dry-run --minimize-conflict-set

# From the install plan we generate a dependency list.
- name: Record dependencies
id: record-deps
run: |
# The tests call out to msys2 commands. We generally do not want to mix toolchains, so
# we are very deliberate about only adding msys64 to the path where absolutely necessary.
${{ (runner.os == 'Windows' && '$env:PATH=("C:\msys64\mingw64\bin;{0}" -f $env:PATH)') || '' }}
cabal build all --dry-run
cat dist-newstyle/cache/plan.json | jq -r '."install-plan"[].id' | sort | uniq > dependencies.txt
- name: Cache Cabal store
uses: actions/cache@v3
# From the dependency list we restore the cached dependencies.
# We use the hash of `dependencies.txt` as part of the cache key because that will be stable
# until the `index-state` values in the `cabal.project` file changes.
- name: Restore cached dependencies
uses: actions/cache/restore@v4
id: cache
with:
path: |
${{ steps.setup-haskell.outputs.cabal-store }}
dist-newstyle
key: cache-${{ env.CABAL_CACHE_VERSION }}-${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles('dependencies.txt') }}

# Now we install the dependencies. If the cache was found and restored in the previous step,
# this should be a no-op, but if the cache key was not found we need to build stuff so we can
# cache it for the next step.
- name: Install dependencies
run: cabal build all --enable-tests --only-dependencies -j --ghc-option=-j4

# Always store the cabal cache.
# This may fail (benign failure) if the cache key is already populated.
- name: Cache Cabal store
uses: actions/cache/save@v4
with:
path: |
${{ steps.setup-haskell.outputs.cabal-store }}
dist-newstyle
key: cache-${{ env.CABAL_CACHE_VERSION }}-${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles('dependencies.txt') }}

# Now we build.
- name: Build [testing]
run: cabal build all --enable-tests -j --ghc-option=-j4

Expand All @@ -102,7 +121,11 @@ jobs:
MSYSTEM: MINGW64
run: cabal test all --enable-tests --test-show-details=direct -j1

- uses: actions/upload-artifact@v3
- name: Build strict-checked-vars with invariants
run: cabal build -f+checktvarinvariants -f+checkmvarinvariants strict-checked-vars

- name: Save Artifact
uses: actions/upload-artifact@v3
with:
name: Build & test logs
path: |
Expand All @@ -127,3 +150,9 @@ jobs:
# uses: mxschmitt/action-tmate@v3
# with:
# limit-access-to-actor: true

- name: Cache HLS artifacts (used to speed up GitHub Codespaces bootstrapping)
if: runner.os == 'Linux' && (matrix.ghc == '8.10.7' || matrix.ghc == '9.6.4')
uses: input-output-hk/actions/cache@latest
with:
ghc_version: ${{ matrix.ghc == '8.10.7' && 'ghc810' || matrix.ghc == '9.6.4' && 'ghc96' }}
165 changes: 165 additions & 0 deletions CONTRBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
# Contributing to the Cardano BAse

## Roles and responsibilities

Currently there are two core maintainers:

* [@lehins](https://github.com/lehins)
* [@tdammers](https://github.com/tdammers)

Anything crypto related should be directed at:

* [@iquerejeta](https://github.com/iquerejeta)

People who can help with issues regarding this repository's continuous integration and nix
infrastructure:

* [@angerman](https://github.com/angerman)
* [@hamishmack](https://github.com/hamishmack)

**For security related issues** please consult the security file in the
[Cardano engineering handbook](https://github.com/input-output-hk/cardano-engineering-handbook/blob/main/SECURITY.md).

## Development

We use trunk based developement. Normal development will branch off of master and be
merged back to master.

### Releasing and versioning

Packages from `cardano-base` are released to
[CHaP](https://github.com/input-output-hk/cardano-haskell-packages).

See documentation on the adopted [release and versioning processes](./RELEASING.md) for
more information.

Also see the CHaP README for [instructions](https://github.com/input-output-hk/cardano-haskell-packages#-from-github).

## Building

See the [Readme](https://github.com/input-output-hk/cardano-base#building) for
instructions on building.

## Updating dependencies

Our Haskell packages come from two package repositories:
- Hackage
- [CHaP](https://github.com/input-output-hk/cardano-haskell-packages) (which is
another alternative Hackage from Cardano)

The `index-state` of each repository is pinned to a particular time in `cabal.project`.
This tells Cabal to treat the repository "as if" it was the specified time, ensuring
reproducibility. If you want to use a package version from repository X which was added
after the pinned index state time, you need to bump the index state for X. This is not a
big deal, since all it does is change what packages `cabal` considers to be available when
solving, but it will change what package versions cabal picks for the plan, and so
will likely result in significant recompilation, and potentially some breakage. That
typically just means that we need to fix the breakage (and add a lower-bound on the
problematic package), or add an upper-bound on the problematic package.

Note that `cabal` itself keeps track of what index states it knows about, so when you bump
the pinned index state you may need to call `cabal update` in order for `cabal` to be happy.

The Nix code which builds our packages also cares about the index state. This is
represented by inputs managed by `nix flake`: You can update these by running:
- `nix flake lock --update-input haskellNix/hackage` for Hackage
- `nix flake lock --update-input CHaP` for CHaP

If you fail to do this you may get an error like this from Nix:
```
error: Unknown index-state 2021-08-08T00:00:00Z, the latest index-state I know about is 2021-08-06T00:00:00Z. You may need to update to a newer hackage.nix.
```

### Use of `source-repository-package`s

We *can* use Cabal's `source-repository-package` mechanism to pull in un-released package
versions. However, we should avoid this. In particular, we cannot release
our packages to CHaP while we depend on a `source-repository-package`.

If we are stuck in a situation where we need a long-running fork of a package, we should
release it to CHaP instead (see the [CHaP
README](https://github.com/input-output-hk/cardano-haskell-packages) for more).

If you do add a `source-repository-package`, you need to provide a `--sha256` comment in `cabal.project` so that Nix knows the hash of the content.

## Warnings

While building most compilation warnings will be turned into an error due to
`-Werror` flag. However during development it might be a bit inconvenient thus
can be disabled on per project basis:

```shell
cabal configure <package-name> --ghc-options="-Wwarn"
cabal build <package-name>
```

### Additional documentation

You can find additional documentation on the nix infrastructure used in this
repo in the following places:

- [The haskell.nix user guide](https://github.com/input-output-hk/haskell.nix/blob/documentation/docs/user-guide.md)
- [The nix-tools repository](https://github.com/input-output-hk/nix-tools)
- [The iohk-nix repository](https://github.com/input-output-hk/iohk-nix)

Note that the user guide linked above is incomplete and does not correctly refer
to projects built using `iohk-nix`, as this one is. A certain amount of trial
and error may be required to make substantive changes!

## Working Conventions

### Code formatting

Very soon we will start using [`fourmolu`](https://github.com/fourmolu/fourmolu) for
formatting, but for now a rule of thumb is to follow whatever format is in a module that
is being modified. There is a script
[here](https://github.com/input-output-hk/cardano-base/blob/master/scripts/fourmolize.sh)
which uses nix to format the appropriate directories.

### Compiler warnings

The CI builds Haskell code with `-Werror`, so will fail if there are any compiler warnings.

A particular warning can be turned off, if there is a compelling enough reason to do so,
but it should be done at the module level, rather than for a whole package.

### Commit messages

Summarize changes in around 72 characters or less.

Provide more detailed explanatory text, if necessary. Wrap it to about 72 characters or
so. In some contexts, the first line is treated as the subject of the commit and the rest
of the text as the body. The blank line separating the summary from the body is critical
(unless you omit the body entirely); various tools like `log`, `shortlog` and `rebase` can
get confused if you run the two together.

Explain the problem that this commit is solving, and use one commit per conceptual change.
Focus on why you are making this change as opposed to how (the code explains that). Are
there side effects or other unintuitive consequences of this change? Here's the place to
explain them.

If you use an issue tracker, put references to them at the bottom, like this:

Resolves: #123
See also: #456, #789

### Commit signing

Commits are required to be [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits).

### Pull Requests

We require linear history in `master`, so every PR must be rebase on `master` before it
can be merged. There is a convenience button on a PR "Update branch", but make sure to
select "Update with Rebase" from the drop down.

Keep commits to a single logical change where possible. The reviewer will be happier, and
you’ll be happier if you ever have to revert it. If you can’t do this (say because you
have a huge mess), best to just have one commit with everything in it.

Keep your PRs to a single topic. Including unrelated changes makes things harder for your
reviewers, slowing them down, and makes it harder to integrate new changes.

If you’re working on something that’s likely to conflict with someone else, talk to
them. It’s not a race.

60 changes: 58 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,63 @@ A collection of miscellaneous packages used by Cardano that cover:

Each sub-project has its own README.

Haddock for all packages from master branch can be found here:
[https://cardano-base.cardano.intersectmbo.org](https://cardano-base.cardano.intersectmbo.org/)

All releases for packages found in this repository are recorded in [Cardano Haskell
package repository](https://github.com/input-output-hk/cardano-haskell-packages)
package repository](https://github.com/intersectmbo/cardano-haskell-packages)

## Building

### With `nix`

With nix it is as easy as:

```
$ nix develop
...
$ cabal build all
```

### Without `nix`

Crypotgraphic depencencies needed for building Haskell packages:

* [`libsodium`](https://github.com/jedisct1/libsodium)
* [`libsecp256k1`](https://github.com/bitcoin-core/secp256k1)
* [`libblst`](https://github.com/supranational/blst)

We provide packaged versions for common Operating Systems for all of the above
dependencies: [Download](https://github.com/input-output-hk/iohk-nix/releases/latest)


## GHC

Default version of GHC used in `nix` is `9.2.7`, but we do support other GHC versions
`8.10.7` and `9.6.1`.


### Testing

This is a command to run test suites for all packages:

```
$ cabal build all
```

The test suites use [Tasty](https://github.com/feuerbach/tasty),
which allows for running specific tests.
This is done by passing the `-p` flag to the test program, followed by an `awk` pattern.
You can alternatively use the `TASTY_PATTERN` environment variable with a pattern.
For example, the `cardano-crypto-tests` can be run with:

```shell
$ cabal test cardano-crypto-tests --test-options '-p blake2b_256'
```

or

```shell
$ TASTY_PATTERN="blake2b_256" cabal test cardano-crypto-tests
```

See the [wiki](https://github.com/input-output-hk/cardano-base/wiki) for more documentation.
Loading

0 comments on commit d13b6d6

Please sign in to comment.