Add flake.nix and other files #1231

doronbehar · 2021-09-17T09:56:48Z

Use flake-compat for shell.nix and default.nix compatibility.

~~Currently nix build -L doesn't work, due to the fonts not available in the source, and they can't be downloaded in the sandbox.~~

flake.nix

akavel · 2021-09-17T10:53:25Z

Hm; if nix build doesn't work with this flake, could you help me understand how can I actually use it? I'm not a super experienced Nix user, and not sure what to do now to take advantage of having this flake.nix after I pulled this PR locally :/ should I use the nix shell command with it? but this again seemed to fail with (same Nix error regardless if I do the submodule update command):

$ git submodule update --init --recursive --remote
Submodule 'libtexpdf' (https://github.com/sile-typesetter/libtexpdf.git) registered for path 'libtexpdf'
Cloning into '/home/akavel/prog/sile/libtexpdf'...
Submodule path 'libtexpdf': checked out 'c4ff429664607e1f055d29aeb5dc5d4b6ddb2499'

$ ls -1 libtexpdf/ | wc -l
133

$ nix shell
error: builder for '/nix/store/fy27raxwrfmldax4n0i9rv48gdp8549b-sile-0.11.1-flake.drv' failed with exit code 2;
       last 10 log lines:
       > config.status: creating sile
       > config.status: executing depfiles commands
       > config.status: executing libtool commands
       > Making all in libtexpdf
       > make[2]: Entering directory '/build/s95x4z6kk2vw0nabr92mn7ihb45qp0c5-source/libtexpdf'
       > make[2]: *** No rule to make target 'all'.  Stop.
       > make[2]: Leaving directory '/build/s95x4z6kk2vw0nabr92mn7ihb45qp0c5-source/libtexpdf'
       > make[1]: *** [Makefile:772: all-recursive] Error 1
       > make[1]: Leaving directory '/build/s95x4z6kk2vw0nabr92mn7ihb45qp0c5-source'
       > make: *** [Makefile:501: all] Error 2
       For full logs, run 'nix log /nix/store/fy27raxwrfmldax4n0i9rv48gdp8549b-sile-0.11.1-flake.drv'.

flake.nix

alerque · 2021-09-17T12:21:21Z

I haven't had time to review this in detail, but thanks for the contribution!

Currently nix build -L doesn't work, due to the fonts not available in the source, and they can't be downloaded in the sandbox.

A quick note on fonts, there is already a gentium nixpkg which should be the only font needed to build and run SILE out of the box. The other two sets of fonts are for building the manual (which is a time consuming process that does not need to be done for most SILE builds) and for test fixtures (also not needed if what you want is to run SILE). A third group for building examples is being removed from this repo shortly anyway (see #1220).

If anybody is doing development work on SILE itself I would expect them to have the source checked out anyway and be able to do the font downloads on the host system on in a nix shell (with curl). They should not get in the way of running a dev version.

doronbehar · 2021-09-17T13:18:55Z

@akavel I'm not sure I'm more experienced in Nix then you :) That FONTCONFIG_FILE was a good piece missing (excuse me for not reading thoroughly your PR and noticing that the build in your version works). I haven't added that attribute though, because I disabled the build of the manual, so no fonts are needed at build at all.

The current usages of this flake are:

Test build that is similar to what would happen in Nixpkgs, with nix build -L.
Enter a development environment shell with nix develop.
Allow Nix users to test drive the software easily with nix run github:sile-typesetter/sile.

Feature 3 doesn't work, due to the Git Submodule issue, but we can test it once we decide how to tackle it with nix run github:doronbehar/sile/nixFlake.

@akavel nix shell failed for you righteously because the build was broken. nix shell is meant for entering a shell where the program sile is available. Perhaps now it will work for you, with the build working? I forced pushed.

doronbehar · 2021-09-17T13:44:56Z

Some ideas for tackling libtexpdf:

Add it as a flake input (pros: easy handling inside this flake.nix) but it might go out of sync with the submodule.
Add it as a package to nixpkgs and add it as a buildInput. Might go out of sync with the submodule as well..
Add it with fetchFromGitHub inside this flake.nix like in feat(build): Add flake.nix #1223 but then the hash there would have to be updated every time it's version is bumped.

akavel · 2021-09-17T14:53:14Z

@doronbehar I'm afraid nix build still doesn't work for me 😢 :

$ git fetch origin pull/1231/head:pr1231.02
$ git checkout pr1231.02
$ nix build
error: builder for '/nix/store/v96hg6vzn6pd3xnivkv7hr4ivacgpbqp-sile-0.11.1-flake.drv' failed with exit code 2;
       last 10 log lines:
       > config.status: creating sile
       > config.status: executing depfiles commands
       > config.status: executing libtool commands
       > Making all in libtexpdf
       > make[2]: Entering directory '/build/xblbbmm57jdrjb7va26va9ggdblzp19a-source/libtexpdf'
       > make[2]: *** No rule to make target 'all'.  Stop.
       > make[2]: Leaving directory '/build/xblbbmm57jdrjb7va26va9ggdblzp19a-source/libtexpdf'
       > make[1]: *** [Makefile:772: all-recursive] Error 1
       > make[1]: Leaving directory '/build/xblbbmm57jdrjb7va26va9ggdblzp19a-source'
       > make: *** [Makefile:501: all] Error 2
       For full logs, run 'nix log /nix/store/v96hg6vzn6pd3xnivkv7hr4ivacgpbqp-sile-0.11.1-flake.drv'.
$ git describe
v0.11.1-141-gd0174488
$ ls -1 libtexpdf/ | wc -l
133

alerque · 2021-09-17T14:56:07Z

Of the 3 options there, ② makes the most sense long term. We're planning on splitting it out with it's own release cycle anyway. I've already setup the Arch Linux packaging getting it ready for a split for that to be it's own library package. Right now we always have it pinned to the master branch HEAD, but as soon as we start tagging releases there it will just be pinned to whatever the latest tag is.

Option 3 is also viable. I have a make job to confirm the submodule is up to date, I can have it update the hash in the flake too.

doronbehar · 2021-09-17T17:24:51Z

Of the 3 options there, ② makes the most sense long term. We're planning on splitting it out with it's own release cycle anyway. I've already setup the Arch Linux packaging getting it ready for a split for that to be it's own library package. Right now we always have it pinned to the master branch HEAD, but as soon as we start tagging releases there it will just be pinned to whatever the latest tag is.

Option 3 is also viable. I have a make job to confirm the submodule is up to date, I can have it update the hash in the flake too.

Perhaps it will be convenient to have a libtexpdf as a submodule for now, or perhaps forever, so it'd be easier to test changes in that library while developing? I added a TODO in the code. Now, using builtins.fetchGit, this works for me:

nix run -L --refresh github:doronbehar/sile/nixFlake

And also the README is updated with a similar command

And maintaining this flake.nix only requires making sure that the file ./libtexpdf.git-rev has the same git rev as the submodule. No need to update a sha256, as builtins.fetchGit is used, and not fetchFromGitHub.

@akavel I hope now it will work for you 🙏 .

doronbehar

One thing that still bothers me @alerque is that when I nix run . -- --version I get:

SILE vUNKNOWN (Lua 5.2)

In which part of the build would it be correct to write .version or .tarball-version so that it'd be picked up by the build?

alerque · 2021-09-17T17:45:22Z

In which part of the build would it be correct to write .version or .tarball-version so that it'd be picked up by the build?

Does the build environment see the rest of the repository (i.e. the .git dir)? If so then something else is wrong. If not then I think you'll need to pretend to be a source-dist by stuffing the version in .tarball-version. If you skip ahead to .version (which is where it eventually gets used from) the configure/make process is likely to blow it away.

Or if needed I can come up with another mechanism.

doronbehar · 2021-09-17T17:57:00Z

In which part of the build would it be correct to write .version or .tarball-version so that it'd be picked up by the build?

Does the build environment see the rest of the repository (i.e. the .git dir)?

No. Putting the version in .tarball-version worked.

alerque · 2021-09-17T19:11:21Z

I'm having trouble testing this locally too. Interestingly I can use nix run github:doronbehar/sile/nixFlake and it seems to run great. What I can't seem to do is any incantation to run it from the Git working directory:

nix run .
/nix/store/g786j9lkivd7fida2hbz6gncvl1mk4r7-lua-5.2.4/bin/lua: error loading module 'justenoughicu' from file './core/justenoughicu.so':
        libicuio.so.69: cannot open shared object file: No such file or directory
stack traceback:
        [C]: in ?
        [C]: in function 'require'
        ./core/utilities.lua:496: in main chunk
        [C]: in function 'require'
        ./core/sile.lua:40: in main chunk
        [C]: in function 'require'
        ...a272cdbvayq834axmlq1439zhkirl-sile-0.11.1-flake/bin/sile:36: in main chunk
        [C]: in ?

Any idea what's up with that? It seems like they should be the same thing since my local branch is what is in this PR branch right now.

flake.nix

doronbehar · 2021-09-18T10:04:19Z

Any idea what's up with that? It seems like they should be the same thing since my local branch is what is in this PR branch right now

No idea :/ Did you try perhaps using nix run --refresh .?

akavel · 2021-09-18T10:11:08Z

Woohoo @doronbehar @alerque , worked for me now!!! 🎉 ❤️

Both nix build and nix run . were successful on my machine; weird that it didn't work for @alerque :/ could some impurity find its way into the build? 🤔 sounds ugly, given the whole point of Nix is to not have situations like this :/

akavel · 2021-09-18T10:33:30Z

@doronbehar oh, sweet, the self ? rev trick is so awesome!!! ❤️

$ nix run .
warning: Git tree '/home/akavel/prog/sile' is dirty
SILE v0.11.1-dirty-flake (Lua 5.2)
> 

$ git stash
Saved working directory and index state WIP on pr1231.04: 789be0b4 fixup! feat(tooling): Enable use as a Nix flake

$ nix run .
SILE v0.11.1-789be0b4-flake (Lua 5.2)
>

alerque · 2021-09-18T14:08:30Z

No idea :/ Did you try perhaps using nix run --refresh .?

Yes I tried that.

Playing with it some more now, it seems that the trouble is the working directory I was in has a built version of SILE — built on my host system, Arch Linux. If I blow that away with git clean¹ then a then nix run . works fine. For some reason if it finds the built stuff in the current repository it tries to use them, and of course they are a binary miss-match for the system. I presume this is actually SILE's fault for looking at the CWD first, system later.

¹ _{git clean -dxff -e node_modules -e .fonts -e .sources -e .husky}

- Use flake-compat for shell.nix and default.nix compatibility. - Use builtins.fetchGit to fetch libtexpdf instead of relying on the submodule.

doronbehar · 2021-09-18T14:33:07Z

flake.nix

+        src = builtins.filterSource
+          (path: type: type != "directory" || baseNameOf path != ".git")
+        ./.;


git clean -dxff -e node_modules -e .fonts -e .sources -e .husky

Perhaps we should filter these files as well right here?

I don't think that will help, if $CWD is still set to the project directory when it is actually run it won't matter what was or wasn't copied into the Nix build, SILE will still prefer local resources to it's distribution ones. This is intentional so that users can override virtually anything in the system, but it does make the assumption that any resources in $CWD should be usable.

In any case this is a developer only problem and I'm not too worried about it. I've scheduled this for merge so I can test it out on the SILE website before tagging a release. We can issue any adjustments for more developer convenience in a follow-up.

As @ctrlcctrlv noted, this is really the same as #1212. I'm not sure what the solution should even be (maybe nothing), but lets move discussion there.

ctrlcctrlv · 2021-09-18T15:07:00Z

I presume this is actually SILE's fault for looking at the CWD first, system later.

That is also the root cause of #1212, but just affecting you this time and not me. 😆

doronbehar · 2021-09-21T09:21:34Z

One thing that bothers me, is that the src is changed a lot, causing rebuilds to be frequent, even with changes such as adding a comment to flake.nix, even with the current filterSource. I asked a question at https://discourse.nixos.org/t/how-to-make-src-in-a-flake-nix-not-change-a-lot/15129

doronbehar mentioned this pull request Sep 17, 2021

feat(build): Add flake.nix #1223

Closed

4 tasks

akavel reviewed Sep 17, 2021

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

akavel reviewed Sep 17, 2021

View reviewed changes

flake.nix Show resolved Hide resolved

alerque added the enhancement Software improvement or feature request label Sep 17, 2021

doronbehar force-pushed the nixFlake branch from 33ce75f to d017448 Compare September 17, 2021 13:12

doronbehar force-pushed the nixFlake branch from d017448 to 617b797 Compare September 17, 2021 17:20

doronbehar commented Sep 17, 2021

View reviewed changes

alerque added this to the v0.11.2 milestone Sep 17, 2021

doronbehar force-pushed the nixFlake branch from 617b797 to a77a6e3 Compare September 17, 2021 17:56

alerque force-pushed the nixFlake branch 2 times, most recently from 0c37423 to 8935c52 Compare September 17, 2021 18:42

alerque reviewed Sep 17, 2021

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

alerque mentioned this pull request Sep 18, 2021

Package SILE with a different Lua interpreter NixOS/nixpkgs#138420

Closed

doronbehar added 2 commits September 18, 2021 17:10

feat(tooling): Enable use as a Nix flake

8b503bb

- Use flake-compat for shell.nix and default.nix compatibility. - Use builtins.fetchGit to fetch libtexpdf instead of relying on the submodule.

docs(readme): Add note about using via Nix flake

fe7637a

alerque force-pushed the nixFlake branch from 789be0b to c3e2e9d Compare September 18, 2021 14:10

alerque enabled auto-merge September 18, 2021 14:10

doronbehar commented Sep 18, 2021

View reviewed changes

alerque added 2 commits September 18, 2021 17:58

docs(manual): Add note about using via Nix flake

dd1e149

chore(build): Inform Nix flake about current libtexpdf version

4dfa5e7

alerque force-pushed the nixFlake branch from c3e2e9d to 5f1d188 Compare September 18, 2021 14:59

alerque disabled auto-merge September 18, 2021 15:05

chore(build): Include Nix tooling in source distribution

cb6df6c

alerque force-pushed the nixFlake branch from 5f1d188 to cb6df6c Compare September 18, 2021 15:19

alerque enabled auto-merge September 18, 2021 15:19

alerque merged commit 326afbd into sile-typesetter:master Sep 18, 2021

This was referenced Sep 18, 2021

Docker image siletypesetter/sile:latest seems broken? #1212

Closed

Test Nix functionality #1238

Closed

doronbehar mentioned this pull request Sep 29, 2021

feat(nix): Ignore many files when creating src #1255

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add flake.nix and other files #1231

Add flake.nix and other files #1231

doronbehar commented Sep 17, 2021 •

edited

Loading

akavel commented Sep 17, 2021 •

edited

Loading

alerque commented Sep 17, 2021

doronbehar commented Sep 17, 2021

doronbehar commented Sep 17, 2021

akavel commented Sep 17, 2021 •

edited

Loading

alerque commented Sep 17, 2021 •

edited

Loading

doronbehar commented Sep 17, 2021

doronbehar left a comment

alerque commented Sep 17, 2021

doronbehar commented Sep 17, 2021

alerque commented Sep 17, 2021

doronbehar commented Sep 18, 2021

akavel commented Sep 18, 2021 •

edited

Loading

akavel commented Sep 18, 2021 •

edited

Loading

alerque commented Sep 18, 2021

doronbehar Sep 18, 2021

alerque Sep 18, 2021

alerque Sep 18, 2021

ctrlcctrlv commented Sep 18, 2021

doronbehar commented Sep 21, 2021

Add flake.nix and other files #1231

Add flake.nix and other files #1231

Conversation

doronbehar commented Sep 17, 2021 • edited Loading

akavel commented Sep 17, 2021 • edited Loading

alerque commented Sep 17, 2021

doronbehar commented Sep 17, 2021

doronbehar commented Sep 17, 2021

akavel commented Sep 17, 2021 • edited Loading

alerque commented Sep 17, 2021 • edited Loading

doronbehar commented Sep 17, 2021

doronbehar left a comment

Choose a reason for hiding this comment

alerque commented Sep 17, 2021

doronbehar commented Sep 17, 2021

alerque commented Sep 17, 2021

doronbehar commented Sep 18, 2021

akavel commented Sep 18, 2021 • edited Loading

akavel commented Sep 18, 2021 • edited Loading

alerque commented Sep 18, 2021

doronbehar Sep 18, 2021

Choose a reason for hiding this comment

alerque Sep 18, 2021

Choose a reason for hiding this comment

alerque Sep 18, 2021

Choose a reason for hiding this comment

ctrlcctrlv commented Sep 18, 2021

doronbehar commented Sep 21, 2021

doronbehar commented Sep 17, 2021 •

edited

Loading

akavel commented Sep 17, 2021 •

edited

Loading

akavel commented Sep 17, 2021 •

edited

Loading

alerque commented Sep 17, 2021 •

edited

Loading

akavel commented Sep 18, 2021 •

edited

Loading

akavel commented Sep 18, 2021 •

edited

Loading