Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: add a meson build configuration #1910

Open
wants to merge 36 commits into
base: main
Choose a base branch
from

Conversation

oscarbenjamin
Copy link
Contributor

This is incomplete but I am putting it here as a talking point:

Would there be any interest in flint using meson as a build system or as an alternative experimental build system?

For downstream usecases it would be nicer if Flint used something like meson rather than autotools et al. For python-flint for example if flint used either cmake or meson then it could be incorporated as a meson subproject after flintlib/python-flint#129 is merged.

From my limited testing it seems faster to build Flint with meson+ninja rather than configure+make. Currently though the meson configuration here does not build on all platforms and it is probably compiling things differently so any timing comparison is not meaningful at this stage.

Another potential advantage of using meson could be using its (experimental) SIMD module which facilitates using runtime detection for things like AVX:
https://mesonbuild.com/Simd-module.html
Runtime detection of CPU capabilities is what would be needed for python-flint to leverage AVX because it is not possible for Python wheel tags to be more fine-grained than e.g. x86_64.

The meson configuration code shown here is incomplete because it does not handle options and things and also it is deliberately written in the simplest possible way.

To test building this with meson first install meson and ninja (e.g. pip install meson ninja) and then:

./bootstrap.sh
./configure
meson setup build
meson compile -C build

You can then install with

meson install -C build

Arguments like --prefix should be passed to meson setup.

A usable implementation of building with meson would need to work without the bootstrap/configure steps. I cannot personally turn the changes in this PR into a complete replacement of configure etc but I could turn it into a usable experimental alternative.

Another possibility is that I could submit the same files to meson's wrapdb database so that other projects can build flint using meson without the meson build files being part of the flint source tree:
https://mesonbuild.com/Wrapdb-projects.html

@oscarbenjamin
Copy link
Contributor Author

To be clear: the build here does not work. In Linux at least I get a completed build but some functions are missing. Probably not much is needed to get to a basic working state though.

@fredrik-johansson
Copy link
Collaborator

Maybe as a first step this can be brought up to parity with and replace CMake. I'd rather not have three parallel build systems.

To replace our autotools setup, we'd need to support all its capabilities and someone would have to commit to maintaining the build system in the medium term.

@albinahlback
Copy link
Collaborator

Would there be any interest in flint using meson as a build system or as an alternative experimental build system?

If we change build systems at all, I think they should be replaced. In other words, Meson would replace both CMake and Autotools. To answer your question, yes, under the right circumstances.

For downstream usecases it would be nicer if Flint used something like meson rather than autotools et al. For python-flint for example if flint used either cmake or meson then it could be incorporated as a meson subproject after flintlib/python-flint#129 is merged.

Apart from this example, why would it be more nice? I have gotten a lot of people seemingly being afraid of Autotools, but I have never understood why. So why?

From my limited testing it seems faster to build Flint with meson+ninja rather than configure+make.

So the most important things here: What was the difference, and what C flags where you using in each case?

Another potential advantage of using meson could be using its (experimental) SIMD module which facilitates using runtime detection for things like AVX: https://mesonbuild.com/Simd-module.html Runtime detection of CPU capabilities is what would be needed for python-flint to leverage AVX because it is not possible for Python wheel tags to be more fine-grained than e.g. x86_64.

I mean, we can establish run-time detection of CPU's. It is not hard. For x86, we can simply utilize the cpuid instruction which tells us everything we need to know. For ARM we can do something similar, but instead rely on the things in config/config.guess. The harder part is to nicely incorporating it into the runtime environment, I think. Please correct me if I'm wrong.

@albinahlback
Copy link
Collaborator

albinahlback commented Apr 8, 2024

I could maintain a Meson build system if a PR was made that such both Autotools and CMake could be removed. Hence, the functionalities of Meson should include:

  • Detecting the CPU during configuration and allow for cross-compilation,
  • Setting the right assembly routines and parameters,
  • Checking which CFLAGS are preferred for which CPU, and then checking which are supported, and setting them accordingly,
  • A nice way to display all configure options for FLINT,
  • Checking for GMP internals,
  • Checking if fft_small is available,
  • Checking for the correct assembly options, such as if symbols should be prefixed with an (additional) underscore or not.

@oscarbenjamin
Copy link
Contributor Author

I could maintain a Meson build system if a PR was made that such both Autotools and CMake could be removed. Hence, the functionalities of Meson should include:

All these things will be doable somehow. I will check what looks like a good way though...

I'll see if I can actually get a working build first though :)

Apart from this example, why would it be more nice? I have gotten a lot of people seemingly being afraid of Autotools, but I have never understood why. So why?

From python-flint's perspective there are two primary advantages that would come from Flint using either meson or cmake rather than autotools. The same considerations would apply to many other downstream projects as well though:

  1. Portability, particularly for Windows. The fact that flint uses autotools pretty much forces downstream build systems to operate in a unix shell with GNU tools which means using e.g. MinGW on Windows rather than the native tools and the MSVC compiler. I assume that the cmake configuration in Flint now was added to support building Flint with MSVC in conda-forge. Even if on unix it would be better to have something other than shell scripts to manage dependency building.

  2. Integration with other build systems. Build systems like meson support building dependencies as subprojects which means that you can have a large stack of dependencies all tracked from git or from release versions. Then effectively make rebuilds your whole stack including all dependencies but keeping track of incremental rebuilds rather than starting from scratch. In the case of meson this works if the dependency uses either meson or cmake. There is some support for autotools-style dependencies but it is discouraged and after playing around with it I think it can only work in very simple cases and generally does not work well.

Projects that use autotools don't work properly as subprojects because they are opaque from the outside and cannot keep track automatically of when it is necessary to rerun configure or make clean etc. Relatedly I recently needed to bisect a bug in CPython but each time git checked out a new commit I couldn't just run make to get a clean rebuild because too many other things were changing. A usable subproject needs to be able to build cleanly after any changes like git checkout v3.0.1 etc but the only way to ensure a clean build with autotools is to wipe everything and start from scratch.

With meson a rebuild can keep track of when there are changes to the meson.build files or anything else or when built files should be removed, when dependencies have been updated to a new version etc. This is like if make could understand when the Makefile or configure scripts were changed and could know whether anything set or checked during configure changed but could know intelligently how to rebuild without just doing make clean or make -B.

Not all projects can use e.g. meson as their primary build system because build system tooling creates a bootstrapping problem. For example GMP and MPFR are dependencies of gcc itself. Likewise Python is a dependency of meson so it would be awkward for CPython to use meson as its own build system. A solution to that from python-flint's perspective is that we can add external wrapdb files for GMP and MPFR (other people want this as well mesonbuild/wrapdb#97).

In the case of Flint there is no bootstrapping issue though and the ideal solution for python-flint would be if Flint used either meson or cmake. If Flint used meson then it could also track GMP and MPFR as subprojects using the wrap file mechanism. In fact I think if Flint did that then it would not be necessary for python-flint or other downstream projects to do it explicitly because GMP and MPFR could just become recursive subprojects from python-flint's perspective (I haven't tested this).

@oscarbenjamin
Copy link
Contributor Author

I'll see if I can actually get a working build first though :)

Currently I get:

    from .pyflint import *
  File "src/flint/pyflint.pyx", line 1, in init flint.pyflint
    """
ImportError: ./src/flint/types/fmpz.cpython-312-x86_64-linux-gnu.so:
undefined symbol: partitions_fmpz_fmpz

I guess some symbols are not being exported correctly.

@albinahlback
Copy link
Collaborator

Fair points. Let me know if you need help interpreting configure.ac and friends.

@oscarbenjamin
Copy link
Contributor Author

I'll see if I can actually get a working build first though :)

I guess some symbols are not being exported correctly.

Actually this does give a working build I think at least on Linux. I was just linking python-flint against the wrong libflint.so.

The fix is that you have to actually install libflint.so after building it!

diff --git a/meson.build b/meson.build
index ae7cd95a1..1169a801b 100644
--- a/meson.build
+++ b/meson.build
@@ -28,6 +28,7 @@ install_headers(headers_all, subdir: 'flint')
 libflint = library('flint', c_files_all,
   dependencies: flint_deps,
   include_directories: include_directories('src'),
+  install: true
 )
 
 pkg_mod = import('pkgconfig')

@oscarbenjamin
Copy link
Contributor Author

From my limited testing it seems faster to build Flint with meson+ninja rather than configure+make.

So the most important things here: What was the difference, and what C flags where you using in each case?

I'm not explicitly setting any flags so just the default ones are being used.

This is how the meson build compiles the C files by default:

cc -Ilibflint.so.p -I. -I.. -Isrc -I../src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O0 -g -fPIC -MD -MQ libflint.so.p/src_acb_dirichlet_turing_method_bound.c.o -MF libflint.so.p/src_acb_dirichlet_turing_method_bound.c.o.d -o libflint.so.p/src_acb_dirichlet_turing_method_bound.c.o -c ../src/acb_dirichlet/turing_method_bound.c

This is how make compiles them

cc -fPIC -DPIC -march=ivybridge -Wno-stringop-overflow -Wno-stringop-overread -Wall -Werror=implicit-function-declaration -std=c11 -pedantic -O3 -g -funroll-loops -I./src -DBUILDING_FLINT -DFLINT_NOSTDIO -DFLINT_NOSTDARG -c src/ulong_extras/is_square.c -o build/ulong_extras/is_square.lo -MMD -MP -MF build/ulong_extras/is_square.lo.d

The main difference for build time is -O0 vs -O3. Apparently meson's default is -O0 but we can add a different default. I've set -O3 as the default by making release the default for buildtype.

After fixing it to use -O3 I get similar times from both for a full rebuild. The only advantage of meson+ninja in that case is that you don't need to set -j N+1 because it is done automatically by ninja.

For a no-op build i.e. running make when there is nothing to do meson compile -C build takes about 1 second on this slow machine vs 5 seconds for make. If I touch src/gr_mpoly.h then both meson and it's 6 seconds vs 7 seconds to rebuild. For touching a single .c file its 3 seconds for meson vs 6 seconds for make. I'm not using ccache here which would affect these timings.

Probably basic build speed like this is not a significant factor here. What is more important is that it is a more reproducible build each time and that the reproducibility does not come at the cost of slower builds. You would get a big gain in speed in situations where you would otherwise need to use make clean for example because that shouldn't be needed with meson.

Also you can configure multiple build directories differently like:

meson setup release-dir -Dbuildtype=release
meson setup debug-dir -Dbuildtype=debug

Now you can have incremental rebuilds in each differently configured build directory without them clobbering each other:

meson compile -C release-dir
meson compile -C debug-dir

With autotools after ./configure --foo=bar you would have to rebuild everything even if you have configured with the exact same options as before and nothing has actually changed.

@albinahlback
Copy link
Collaborator

Yeah, I really like the rebuilding times you presented.

Could you just present the exact building times you get from Meson vs Make for building the whole library (not including the configuration process)? You can go with CFLAGS="-O0" for simplicity. I just want to make sure that building with Meson for one-time builds are not 30% slower or something.

@albinahlback
Copy link
Collaborator

And I just want to make it clear that I will not merge something that does not superset-ish what Autotools currently does. I know this is somewhat of a big task to take on, just so that you are aware of my requirements. Make sure that you exactly know what the Autotools stuff is doing.

And, once more, I am here to help you with the interpretation of the Autoconf stuff.

@oscarbenjamin
Copy link
Contributor Author

These are timings from a fresh git clone using this PR as it stands. The timings are on a relatively slow Linux machine with 4 cores and build using Ubuntu's stock gcc in both cases. Both builds give a usable libflint but only because this machine does not have whatever is needed for fft_small. In both cases I am using the default compiler flags which means -O3. For the timings I am running them in separate directories for meson vs make and they are not sharing anything in terms of the build.

This is the timing for a full rebuild in a fresh clone with make:

$  git clone https://github.com/oscarbenjamin/flint.git
$  cd flint/
$  git checkout pr_meson
$  ./bootstrap.sh 
$  ./configure 
$  time make -j5 > make-output.txt
...
real	5m20.621s
user	18m48.493s
sys	2m8.120s

This is the full rebuild timing with meson+ninja:

$  git clone https://github.com/oscarbenjamin/flint.git
$  cd flint
$  git checkout pr_meson
$  ./bootstrap.sh 
$  ./configure 
$  meson setup build
$  time meson compile -C build
...
real	5m43.528s
user	17m48.237s
sys	2m24.565s

In this run meson is 7% slower than make but I've run these both a few times today and seen a lot of variability. Many times meson seemed faster but it all takes a bit too long for me to just run it many times and average. For the quoted timings I was careful not to use the computer while the build ran although the screen did go to sleep briefly during the meson build so I woke it up.

These are warm cache timings for a no-op build like running make after you have just run make:

$ time meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
ninja: no work to do.

real	0m1.329s
user	0m1.035s
sys	0m0.355s
$ time make
make: Nothing to be done for 'all'.

real	0m6.010s
user	0m5.764s
sys	0m0.242s

So in this case meson takes 1.3 seconds to see that nothing needs to be done and make takes 6 seconds.

These are timings after touch src/gr_mpoly.h:

$ touch src/gr_mpoly.h 
$ time make
  CC  gr/mpoly.c
  CC  gr_mpoly/add.c
  CC  gr_mpoly/combine_like_terms.c
  CC  gr_mpoly/equal.c
  CC  gr_mpoly/fit_bits.c
  CC  gr_mpoly/fit_length.c
  CC  gr_mpoly/fit_length_fit_bits.c
  CC  gr_mpoly/fit_length_reset_bits.c
  CC  gr_mpoly/gen.c
  CC  gr_mpoly/get_coeff_scalar_fmpz.c
  CC  gr_mpoly/get_coeff_scalar_ui.c
  CC  gr_mpoly/init.c
  CC  gr_mpoly/inlines.c
  CC  gr_mpoly/is_canonical.c
  CC  gr_mpoly/mul.c
  CC  gr_mpoly/mul_johnson.c
  CC  gr_mpoly/mul_monomial.c
  CC  gr_mpoly/mul_scalar.c
  CC  gr_mpoly/neg.c
  CC  gr_mpoly/push_term.c
  CC  gr_mpoly/randtest_bits.c
  CC  gr_mpoly/randtest_bound.c
  CC  gr_mpoly/set.c
  CC  gr_mpoly/set_coeff_scalar_fmpz.c
  CC  gr_mpoly/set_coeff_scalar_ui.c
  CC  gr_mpoly/set_scalar.c
  CC  gr_mpoly/sort_terms.c
  CC  gr_mpoly/sub.c
  CC  gr_mpoly/write.c
Building libflint.so.20.0.0

real	0m10.885s
user	0m9.677s
sys	0m1.055s
$ touch src/gr_mpoly.h 
$ time meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
[30/30] Linking target libflint.so

real	0m5.194s
user	0m8.241s
sys	0m1.769s

It looks like they both built 30 targets and meson did it in 5 seconds vs 10 seconds for make.

These are timings for touch src/fmpz/fmpz.c:

$ touch src/fmpz/fmpz.c 
$ time make
  CC  fmpz/fmpz.c
Building libflint.so.20.0.0

real	0m6.411s
user	0m5.942s
sys	0m0.467s
$ touch src/fmpz/fmpz.c
$ time meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
[2/2] Linking target libflint.so

real	0m3.550s
user	0m2.434s
sys	0m1.020s

Both built 2 targets (fmpz.o and libflint.so) and it took 3 seconds for meson vs 6 seconds for make.

@oscarbenjamin
Copy link
Contributor Author

These are timings for a full build in a fresh clone but on a faster Mac laptop that has a 12 core M3 CPU. On this computer I have a few local changes meaning that the meson build can get to the end but it fails with a link error, due to some assembly preprocessor defines not being correct. I think that the timings are approximately comparable any way though because linking is only the last step.

This is the timing with make:

$  git clone https://github.com/oscarbenjamin/flint.git
$  cd flint/
$  git checkout pr_meson
$  ./bootstrap.sh 
$  ./configure 
$ time make -j13 > make-output.txt
...
make -j13 > make-output.txt  213.18s user 102.57s system 841% cpu 37.527 total

This is the timing with meson:

$  git clone https://github.com/oscarbenjamin/flint.git
$  cd flint/
$  git checkout pr_meson
$ git apply ../mac-fixes.diff  # some local fixes...
$  ./bootstrap.sh 
$  ./configure 
$ meson setup build
$ time meson compile -C build
...
meson compile -C build  189.25s user 86.31s system 710% cpu 38.780 total

In this case the meson build is about 11% faster than make but the final link step did not complete:

Undefined symbols for architecture arm64:
  "_flint_mpn_mul_10n", referenced from:
      _flint_mpn_mul_func_n_tab in src_mpn_extras_mul_basecase.c.o
  "_flint_mpn_mul_11n", referenced from:
      _flint_mpn_mul_func_n_tab in src_mpn_extras_mul_basecase.c.o
...

The problem is something to with this preprocessor define:

#elif FLINT_HAVE_ASSEMBLY_armv8
mp_limb_t flint_mpn_mul_1n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_2n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_3n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_4n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_5n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_6n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_7n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_8n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_9n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_10n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_11n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_12n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_13n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_14n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
mp_limb_t flint_mpn_mul_15n(mp_ptr, mp_srcptr, mp_srcptr, mp_size_t);
#else

The diff applied to make that get as far as it did was:

diff --git a/meson.build b/meson.build
index 6a73b0dbe..31734a767 100644
--- a/meson.build
+++ b/meson.build
@@ -1,7 +1,10 @@
 project('FLINT', 'c',
   version: '3.1.0',
   license: 'LGPL3+',
-  default_options: ['buildtype=release'],
+  default_options: [
+    'buildtype=release',
+    'c_std=c11',
+  ],
 )

 cc = meson.get_compiler('c')
@@ -12,6 +15,14 @@ mpfr_dep = dependency('mpfr', version: '>= 4.1.0')

 flint_deps = [gmp_dep, mpfr_dep, m_dep]

+add_project_arguments(
+  '-march=armv8-a',
+  '-DBUILDING_FLINT',
+  '-DFLINT_NOSTDIO',
+  '-DFLINT_NOSTDARG',
+  '-Werror=implicit-function-declaration',
+  language: 'c')
+
 subdir('src')

 headers_all = []
diff --git a/src/meson.build b/src/meson.build
index 66c0451ab..2a9d0d4fd 100644
--- a/src/meson.build
+++ b/src/meson.build
@@ -1947,15 +1947,15 @@ c_files_all_subdir = [
   'fft/mul_truncate_sqrt2.c',
   'fft/negmod_2expp1.c',
   'fft/normmod_2expp1.c',
-  #'fft_small/default_ctx.c',
-  #'fft_small/fmpz_poly_mul.c',
-  #'fft_small/mpn_helpers.c',
-  #'fft_small/mpn_mul.c',
-  #'fft_small/mulmod_statisfies_bounds.c',
-  #'fft_small/nmod_poly_mul.c',
-  #'fft_small/sd_fft.c',
-  #'fft_small/sd_fft_ctx.c',
-  #'fft_small/sd_ifft.c',
+  'fft_small/default_ctx.c',
+  'fft_small/fmpz_poly_mul.c',
+  'fft_small/mpn_helpers.c',
+  'fft_small/mpn_mul.c',
+  'fft_small/mulmod_statisfies_bounds.c',
+  'fft_small/nmod_poly_mul.c',
+  'fft_small/sd_fft.c',
+  'fft_small/sd_fft_ctx.c',
+  'fft_small/sd_ifft.c',
   'fft/split_bits.c',
   'fmpq/add.c',
   'fmpq/addmul.c',
@@ -5275,7 +5275,6 @@ c_files_all_subdir = [
   'qsieve/poly.c',
   'qsieve/primes_init.c',
   'qsieve/square_root.c',
-  'test/main.c',
   'thread_pool/clear.c',
   'thread_pool/distribute_work.c',
   'thread_pool/find_work.c',

@albinahlback
Copy link
Collaborator

Timings are looking very good. I think we can afford a small drop in a one-time build while we gain a lot in rebuilds! Thank you for looking into this.

@oscarbenjamin
Copy link
Contributor Author

to help you with the interpretation of the Autoconf stuff.

What would be great is if you could just summarise from a high level what it is that configure is doing to modify the sources and also how that affects the generated Makefile. Right now I am (inadvisedly?) trying to make the meson build work after configure has run so that I don't need to replicate its effect on the sources yet. However it seems that configure has encoded some information into the Makefile in relation to fft_small that I don't immediately understand to be able to replicate in meson.

@albinahlback
Copy link
Collaborator

Regarding the wrong assembly linking: Is this with Autotools or is this with Meson?

@oscarbenjamin
Copy link
Contributor Author

Regarding the wrong assembly linking: Is this with Autotools or is this with Meson?

With meson. It works fine with autotools.

@albinahlback
Copy link
Collaborator

Oh, I see. If it is with Meson then it should be expected (unless you have gotten that far already).

@oscarbenjamin
Copy link
Contributor Author

unless you have gotten that far already

I definitely haven't :)

@albinahlback
Copy link
Collaborator

We (= me = Albin) use M4 to make writing assembly a little bit easier. One such things is making the assembler write the correct symbol. MacOS typically prefix their symbols with an underscore, which we test during the configuration.

@albinahlback
Copy link
Collaborator

So if you write int foo(void) in C, the generated symbol on some systems will actually be _foo and not foo.

@albinahlback
Copy link
Collaborator

There are a couple of other things that is needed to be able to write assembly, but I will try to write a list of what the whole Autotools process is doing.

@albinahlback
Copy link
Collaborator

Note that I will herein only describe the process for Autotools. Any local
functions used in configure.ac can be found in acinclude.m4. Remember that
stuff in configure.ac and acinclude.m4 is taken from GMP, so their copyright
claim should be preserved.

File structure

configure.ac

Contains the main script for the configuration.

acinclude.m4

Contains functions that configure.ac may use.

bootstrap.sh

Generates configure from configure.ac.

config/config.guess and config/configfsf.guess

In an Autotools project, one can push --host=aarch64-pc-linux-gnu to
configure to compile for 64-bit Arm architecture running Linux. In case you do
not specify what host is going to run the library, we run config/config.guess
to guess what host is going to run the library. In this script, we first run
config/configfsf.guess to get a guess, and then we get more precise in
config/config.guess to (try to) get the exact architecture.

config/config.sub and config/configfsf.sub

Validates system triplets.

Makefile.in

Basically Makefile but with some parts that has to be filled in during
configuration.

Layout in configure.ac

I have some titles in configure.ac which I will follow here.

preamble

We start by initializing the project, setting copyrights as well as version
number from the VERSION file. We keep a separate VERSION file to be able to
very easily change the version number, without bloating the Git-history on
configure.ac. We also set the SO-versions (manually?).

initialize libtool

We check if CFLAGS are set. Then we initialize libtool, for which we push the
option to disable static by default. If CFLAGS was not set, we remove any
CFLAGS that libtool may want to push.

build system

We do not allow specifying --target. We could get system triplets here, but
that is already invoked when we initialized libtool.

configure headers

We set which headers we are about to configure.

architecture specifics

We set the default CFLAGS. Then we check if we can recognize any architectures
we especially develop for, which is all x86 architectures supporting the ADX
instruction set as well as all Arm v8. We push -march options as well as what
assembly path and parameter path we should use for the host. Note that we do not
go for assembly when system is Windoze since Windoze for some reason chose to go
with their own ABI, which is incompatible with every other x86 OS.

Now, if we know by now that the processor supports AVX2 or NEON, we enable
fft_small module to reduce the time configuring. Otherwise we will test this
later.

features

Here we set the options available and the helpstrings. We check if we got an
okay value. Remember to preserve the defaults here.

packages

We set which packages one can include. Mandatory ones are GMP and MPFR. We allow
specifying include directory and library directory individually. We also check
that one does not specify both --with-PKG and --with-PKG-lib with
--with-PKG-include.

programs

We check what to use for mkdir -p, which is used in Make. Probably doesn't
have to be checked in Meson.

environment variables

We allow specifying different C flags to the test programs.

check programs and system

Check what keyword is used for defining inlined functions.

We check that the compiler supports -c and -o simultaneously (I think only
very old compilers did not allow this, so you can skip this).

We check for endianness.

We check which OS is being targeted, and set the filenames for the dynamic
library accordingly.

We set the static library name.

We also check how to push an option to the linker. This is usually -Wl. You
may or may not have to check for this, but it is required to test FLINT with the
correct library, so that it tests against the compiled library, and not some
other version that may be installed system-wide or whatever.

We check if the system is strongly-ordered. For now, it suffice to set all x86
systems to this. Some algorithms depend on this, which is why it is important.

check GMP

We check that the GMP version is greater or equal to 6.2.1. We check if we have
long long limbs, and set gmpcompat.h accordingly.

check MPFR

We check that MPFR >= 4.1.0.

check ABI

Some systems support multiple ABIs. Here, we check if ABI was specified. If it
was, then we set the ABI (duh). Else, we check how big GMP_LIMB_BITS is (can
only be 32 or 64).

check headers

We check for some headers. Some are optional, others are mandatory. Some are
mandatory only if some option was specified (such as --with-pthread requires
pthread.h).

check libraries

We save all CFLAGS as they may screw up with detecting if functions actually
exists or not. Then we make sure that the math library, GMP and MPFR can be
found. If user wants GMP internals (which is the default), we check for some GMP
internal functions. We always require some MPFR internals, so that has to be
checked. If user wants pthread, we check how to specify this. We also check
for BLAS and NTL if these are wanted.

Finally, we have an option for checking MPFR, which is maintainer level only. We
want to check for Nemo in a CI, but the MPFR that is built with Julia contains
some sanitiser settings, which gives an error when checking for such functions.
Hence, for this reason, we have this maintainer level option here. I do not know
if you will need it.

check settings and environment

The position independent flag that has to be pushed in order to generate a
shared library gets checked with libtool. We substitute this into the Makefile.

We check for cpu_set_t, which is used for threading stuff.

We check that alloca works as intended. You can skip this check if you want as
I've never seen any modern system not having it.

We check for aligned_alloc and _aligned_malloc. This is very important
downstream as old MacOS do not have either of these functions, although the
compiler state that it is C11 compatible.

CFLAGS

We check if compiler is Clang. If it is, we push an error flag for unknown
options so that it generates an error if we give some faulty option. Otherwise
the flags may be available, but still pushed to Make.

We check if we are going to generate code coverage. If so, we push adequate
flags.

If we are generating debugging information, which is the default, we check that
we actually can do it.

If ABI was set from user, we check that we can compile with that ABI.

We check if user wants AVX2 and AVX512, and set flags accordingly.

If the user did not specify CFLAGS, we check all flags we would like the
compiler to accepts. We also push -funroll-loops to some modules, so make sure
to test if this flag is available. If unrolling is not available, then we do not
push this to Make.

However, if the user specified the flags, make sure that all the flags are
accepted.

If the user specified TESTCFLAGS, we simply push these flags along with the
mandatory flags (such as -g for debugging) to Make. Otherwise, if the user did
not specify them, we use the same flags as for the rest of the repository.

We also set the C preprocessor flags to include src/ for the headers.

fft_small module

We check if the fft_small module is available. If x86, check for immintrin.h
and AVX2, for which the latter will be cached for most users. Else if Arm,
check for arm_neon.h and NEON instructions. Else it is not available. The
answer should be pushed to the Makefile.

Assembly

And now to the good parts!

If assembly is not wanted (then Albin is very disappointed), then we push that
we do not want assembly to the Makefile.

For the rest of this section, we assume that assembly is wanted.

First, we check that we have a path for the assembly code and that the ABI is
64-bit. If not, we send out a warning that we do not have any assembly code
available. Note that Windoze does not have a path for assembly since their ABI
is non-standard.

Now, we initialise the M4 configuration file used for the assembly routines,
called config.m4. We check that the M4 preprocessor is available, that nm
is available, and then we check what syntax to use. Some syntax is specific to
the architecture. Some syntax is architecture specific. We include the correct
M4 file(s) in config.m4 and set a C preprocessor constant according to what
assembly we are using (such as FLINT_HAVE_ASSEMBLY_armv8).

Finally, we push the assembler path to the Makefile.

parameters

We set the correct parameter path for the architecture.

substitutions and definitions

We basically just make substitutions, definitions and linking based on options
and found values.

epilog

We configure Makefile, flint.pc and flint.h according to the values we've
gotten.

@albinahlback
Copy link
Collaborator

albinahlback commented Apr 9, 2024

Things to consider when replacing Makefile

  1. We want to be able to compile with specific CFLAGS for certain modules.
    Unrolling of loops should be made for those listed at around line 100.

  2. We want to be able to compile assembly. This includes generating the actual
    assembly file, which includes both PIC and non-PIC, as well as the actual
    object files.

  3. As stated in the previous post, we want to be able to link test files with
    the generated library, not some installed library.

  4. Source files should (preferably) be added automatically, as well as their
    dependencies. But I'm all ears if they should be added manually. This was a
    requirement from Bill Hart when I first started mixing with the build system,
    but I'm open to have it some other way if it reduces build times or whatever.
    Dependencies, however, should be automated.

  5. Would be very good if one could list variable without altering the files. For
    instance, with make print-VAR we print variable VAR. This is very nice
    for debugging the Makefile.

  6. I want nice printing of the building process. I do not want gcc -flag1 -flag2 ... -flag10000 -c foo.c -o foo.o as that bloats the screen. Although
    I think some information is nice to have, try to avoid long lines here.

  7. I want it to state when it is building the actual library in a nice way, so
    that you know when it is about to be done. I suppose this is built into
    Meson, so I think you should not have to worry about this.

  8. On some systems, the line width for the terminal is limited to some small
    number of characters (perhaps 1000 or something along that magnitude), which
    becomes a problem when FLINT has a lot of files. For this, we need to build
    merged object files. I do not know if you have to deal with this problem.
    This will become apparent in the CI.

  9. Be able to create tests, tunes (?), profilers and example executables. Note
    that this includes NTL, which requires C++.

  10. Being able to run tests.

  11. Being able to run tests only for a set of modules (for the current module we
    do this via make check MOD="mpn_extras fmpz").

  12. Being able to run tests for a set of modules with multiple threads. That is, if I want to test
    mpn_extras and fmpz with N threads, I want it to partition this test
    into at least N jobs running the tests in parallel (perhaps one job takes
    care of the test for fmpz_abs, fmpz_addmul, etc. while another job takes
    care of flint_mpn_mulhigh_n etc.). In other words, all tests should be
    executed in parallel and no test should be duplicated.

  13. Saying when the test has completed running with a zero return value, so that
    one knows that they are completed (so that it did not stall, or did not
    abort or something).

  14. Being able to compile profilers only for a set of modules.

  15. Check examples. See make checkexamples.

  16. Shortcut for running gdb on a test. Currently, this is done via make debug MOD=XXX ARGS=YYY.

  17. Being able to run Valgrind, both on a set of modules, or on the whole
    library.

  18. Being able to capture the code coverage.

  19. Clean the build (opt-in for cleaning everything configurable).

  20. Installing and uninstalling (equally important to installing).

  21. Create a tarball. See make dist.

@albinahlback
Copy link
Collaborator

I don't think I missed anything, but of course I left some (more or less obvious) things unsaid. Let me know if there is something that needs to be explained in more detail.

@oscarbenjamin
Copy link
Contributor Author

Things to consider when replacing Makefile

  1. We want to be able to compile with specific CFLAGS for certain modules.
    Unrolling of loops should be made for those listed at around line 100.

In meson it is not possible to simply set different compiler flags on a per file basis but this kind of case where a few files use different flags can be handled by building them into an intermediate static library that is then linked into the end product:
mesonbuild/meson#1367

  1. We want to be able to compile assembly. This includes generating the actual
    assembly file, which includes both PIC and non-PIC, as well as the actual
    object files.

This is all doable. Some kinds of assembly have explicit support in meson but in general you can use custom targets to run any commands that need to be run if needed.

  1. As stated in the previous post, we want to be able to link test files with
    the generated library, not some installed library.

With meson this is taken care of automatically by the build configuration. Something like:

libflint = library('flint', c_files_all,
  dependencies: flint_deps,
  include_directories: include_directories('src'),
  install: true
)

test_exe = executable('test', 'test.c', link_with: libflint)

It doesn't use environment variables etc except when configuring.

  1. Source files should (preferably) be added automatically, as well as their
    dependencies. But I'm all ears if they should be added manually. This was a
    requirement from Bill Hart when I first started mixing with the build system,
    but I'm open to have it some other way if it reduces build times or whatever.
    Dependencies, however, should be automated.

Adding source files automatically is disallowed in meson. There are workarounds but the intention is that files are listed explicitly and this is what enables it to be fast and reproducible. Dependencies among .c and .h files are determined automatically by ninja but the files themselves must be listed explicitly for this to work. We can easily make a Python script that could update this for all but one or two flint modules (I imagine fft_small or mpn_extras would be exceptions).

  1. Would be very good if one could list variable without altering the files. For
    instance, with make print-VAR we print variable VAR. This is very nice
    for debugging the Makefile.

I won't show the full output but:

$ meson configure build-dir
Core properties:
  Source dir /Users/enojb/work/dev/flint/tmp/flint
  Build dir  /Users/enojb/work/dev/flint/tmp/flint/build

Main project options:

  Core options                             Current Value                              Possible Values                            Description
  --------------                           -------------                              ---------------                            -----------
  auto_features                            auto                                       [enabled, disabled, auto]                  Override value of all 'auto' features
  backend                                  ninja                                      [ninja, vs, vs2010, vs2012, vs2013,        Backend to use
                                                                                       vs2015, vs2017, vs2019, vs2022, xcode,
                                                                                       none]
  buildtype                                release                                    [plain, debug, debugoptimized, release,    Build type to use
                                                                                       minsize, custom]
...

Those are built-in options but Flint-specific custom options would get listed there as well.

  1. I want nice printing of the building process. I do not want gcc -flag1 -flag2 ... -flag10000 -c foo.c -o foo.o as that bloats the screen. Although
    I think some information is nice to have, try to avoid long lines here.

By default it looks like what I showed above:

$ meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
[30/30] Linking target libflint.so

When you actually run that it shows a running counter like [10/30] compiling fmpz.c. If you pass -v then it will output the actual compiler commands one by one.

  1. I want it to state when it is building the actual library in a nice way, so
    that you know when it is about to be done. I suppose this is built into
    Meson, so I think you should not have to worry about this.

It shows

[30/30] Linking target libflint.so

at the final link step. For a full build it's something like [5120/5120] Linking ....

  1. On some systems, the line width for the terminal is limited to some small
    number of characters (perhaps 1000 or something along that magnitude), which
    becomes a problem when FLINT has a lot of files. For this, we need to build
    merged object files. I do not know if you have to deal with this problem.
    This will become apparent in the CI.

This was a meson bug that was fixed specifically because someone was using meson to build Flint:
mesonbuild/meson#7212

  1. Be able to create tests, tunes (?), profilers and example executables. Note
    that this includes NTL, which requires C++.
  2. Being able to run tests.
    ...

These things will all be possible somehow. It might be that some things that are handled in the Makefile would need to be a separate Python script or something.

  1. Clean the build (opt-in for cleaning everything configurable).

The simple way to do this is just rm -r build-dir. You don't actually need to do that though because you can instead just make a new build directory and use that:

$ meson setup new-build-dir
$ meson configure new-build-dir -Dfoo=bar
$ meson compile -C new-build-dir

Note that meson does everything out-of-tree. It only creates/modifies files etc inside the build directory and leaves the source tree itself untouched.

  1. Installing and uninstalling (equally important to installing).

This is just meson uninstall. You need to still have the build directory that was used for installing though and it does not restore any files that were overwritten during install.

@albinahlback
Copy link
Collaborator

In meson it is not possible to simply set different compiler flags on a per file basis but this kind of case where a few files use different flags can be handled by building them into an intermediate static library that is then linked into the end product: mesonbuild/meson#1367

I don't like this at all. Seems like a very basic thing that high performance libraries want to utilize. Hopefully we can do it gracefully.

Adding source files automatically is disallowed in meson. There are workarounds but the intention is that files are listed explicitly and this is what enables it to be fast and reproducible. Dependencies among .c and .h files are determined automatically by ninja but the files themselves must be listed explicitly for this to work. We can easily make a Python script that could update this for all but one or two flint modules (I imagine fft_small or mpn_extras would be exceptions).

Yeah, don't be too worried about this. I'm down for only doing it manually.

  1. On some systems, the line width for the terminal is limited to some small
    number of characters (perhaps 1000 or something along that magnitude), which
    becomes a problem when FLINT has a lot of files. For this, we need to build
    merged object files. I do not know if you have to deal with this problem.
    This will become apparent in the CI.

This was a meson bug that was fixed specifically because someone was using meson to build Flint: mesonbuild/meson#7212

Haha, that's funny! Nice that is was being fixed.

  1. Be able to create tests, tunes (?), profilers and example executables. Note
    that this includes NTL, which requires C++.
  2. Being able to run tests.
    ...

These things will all be possible somehow. It might be that some things that are handled in the Makefile would need to be a separate Python script or something.

Alright. I just want to make it clear that I would like to avoid users having to call something else than meson in order to perform the tests. Additionally, I would really prefer to keep everything in the Meson ecosystem, just like we have contained (almost) everything into Makefile.

Although I have my worries, I think this all sounds good. I'm looking forward to see the progress for this.

@albinahlback
Copy link
Collaborator

And thank you for looking into this!

@oscarbenjamin
Copy link
Contributor Author

  • Being able to run tests.
  • Being able to run tests only for a set of modules (for the current module we
    do this via make check MOD="mpn_extras fmpz").
  • Being able to run tests for a set of modules with multiple threads. That is, if I want to test
    mpn_extras and fmpz with N threads, I want it to partition this test
    into at least N jobs running the tests in parallel (perhaps one job takes
    care of the test for fmpz_abs, fmpz_addmul, etc. while another job takes
    care of flint_mpn_mulhigh_n etc.). In other words, all tests should be
    executed in parallel and no test should be duplicated.
  • Saying when the test has completed running with a zero return value, so that
    one knows that they are completed (so that it did not stall, or did not
    abort or something).

I think all of these things are handled natively by meson's test command:
https://mesonbuild.com/Unit-tests.html

  • meson test runs the tests.
  • By default tests are run in parallel but I think it is separate processes rather than separate threads.
  • You can control how many processes with MESON_TESTTHREADS.
  • Tests to run can be specified by test name or in groups using --suite.
  • I'm sure it will set the return code correctly.
  • Tests can be repeated and timeouts can be set.
  • gdb, valgrind, coverage etc supported.

Testing won't work with the meson.build here yet though because I haven't added any configuration to build the tests.

@oscarbenjamin
Copy link
Contributor Author

There is a meson bug about test executables being built by default:
mesonbuild/meson#2518

A workaround is to have a configure option for building tests:

meson setup build -Dbuild_tests=true
meson test

It should be possible to make it so that meson test errors if build_tests has not been configured.

Comment on lines 412 to 431
cfg_data_internal.set_quoted('LT_OBJDIR', '.libs/',
description: 'Define to the sub-directory where libtool stores uninstalled libraries.')
cfg_data_internal.set_quoted('PACKAGE_BUGREPORT',
'https://github.com/flintlib/flint/issues/',
description: 'Define to the address where bug reports for this package should be sent.')
cfg_data_internal.set_quoted('PACKAGE_NAME', 'FLINT',
description: 'Define to the full name of this package.')
cfg_data_internal.set_quoted('PACKAGE_STRING', 'FLINT ' + FLINT_VERSION_FULL,
description: 'Define to the full name and version of this package.')
cfg_data_internal.set_quoted('PACKAGE_TARNAME', 'flint',
description: 'Define to the one symbol short name of this package.')
cfg_data_internal.set_quoted('PACKAGE_URL', 'https://flintlib.org/',
description: 'Define to the home page for this package.')
cfg_data_internal.set_quoted('PACKAGE_VERSION', FLINT_VERSION_FULL,
description: 'Define to the version of this package.')

cfg_data_internal.set('STDC_HEADERS', 1,
description: '''Define to 1 if all of the C89 standard headers exist (not just the ones
required in a freestanding environment). This macro is provided for
backward compatibility; new code need not use it.''')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You do not need to define these, just to let you know.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the initial attempt to get to a free standing build that does not depend on the bootstrap.sh and the autotools ./configure I tried to reproduce config.h and flint-config.h as exactly as possible so that I could diff them to see what defines were missing/different. I got that to a good enough state though so what is needed now is just ensuring that the logic for choosing the actual defined values is correct and. The unneeded defines can be removed in later cleanup.

Also it seems that configure generates a bunch of HAVE_FOO_H defines that are mostly unused in Flint i.e. these are the only ones actually used:

$ git grep 'HAVE_.*_H'
acinclude.m4:dnl  FLINT_HAVE_FFT_SMALL_ARM_H
acinclude.m4:AC_DEFUN([FLINT_HAVE_FFT_SMALL_ARM_H],
acinclude.m4:dnl  FLINT_HAVE_FFT_SMALL_X86_H
acinclude.m4:AC_DEFUN([FLINT_HAVE_FFT_SMALL_X86_H],
acinclude.m4:AC_REQUIRE([FLINT_HAVE_FFT_SMALL_ARM_H])
acinclude.m4:AC_REQUIRE([FLINT_HAVE_FFT_SMALL_X86_H])
doc/source/history.rst:  * Change ``#ifdef FLINT_HAVE_FFT_SMALL`` to ``#if FLINT_HAVE_FFT_SMALL`` (AA).
src/flint.h.in:#if defined(FLINT_HAVE_VA_LIST) && defined(FLINT_HAVE_FILE)
src/fmpz_lll/is_reduced_d.c:#if HAVE_FENV_H
src/fmpz_lll/is_reduced_d.c:#if HAVE_FENV_H
src/fmpz_lll/is_reduced_d_with_removal.c:#if HAVE_FENV_H
src/fmpz_lll/is_reduced_d_with_removal.c:#if HAVE_FENV_H
src/mpn_extras.h:# define FLINT_HAVE_MUL_FUNC(n, m) FLINT_HAVE_MUL_N_FUNC(n)
src/test/t-io.c:#if HAVE_UNISTD_H && FLINT_COVERAGE
src/thread_pool/init.c:#if FLINT_USES_PTHREAD && defined(HAVE_PTHREAD_NP_H)
src/thread_pool/restore_affinity.c:#if FLINT_USES_PTHREAD && defined(HAVE_PTHREAD_NP_H)
src/thread_pool/set_affinity.c:#if FLINT_USES_PTHREAD && defined(HAVE_PTHREAD_NP_H)

Do the others like HAVE_MALLOC_H etc still need to be included in config.h or flint-config.h?

Is it that the configure is just supposed to fail if e.g. malloc.h is not available?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We require the standard headers stdio.h, stdlib.h, stdint.h, string.h, stdarg.h, math.h, float.h and errno.h. We check for this during the configuration in order to be able to debug easier.

We assume that malloc.h exists for Windows and MinGW, but check if it exists in the configuration as well.

Other things are dependent on other things. fenv.h is used if user has it; same goes with pthread_np.h.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You do not need to generate HAVE_FOO_H for everything, that is. However, I want it to be present in any logs generated, so that it is easier to debug.

@oscarbenjamin
Copy link
Contributor Author

I am intending to continue with this but just busy with other things at work for the time being.

The current state of this is that it can build from a fresh checkout without using any of the autotools stuff. I have added a meson_bootstrap.py that can be used in place of bootstrap.sh. It copies/generates the meson.build files and also generates a configure script and Makefile that wrap calls to meson. In principle you can use it just like:

./meson_bootstrap.py
./configure
make

(This overwrites configure/Makefile so you would need to run bootstrap.sh to get back to autotools after.)

The configure script does not yet accept every possible argument that the autotools one does and the Makefile is missing other targets. Some things are only possible with direct calls to meson.

The things that work are: make, make test, make install, make coverage, make clean. This builds and passes tests (both Flint and python-flint tests) on all of the machines I have tested although some autodetection is missing so I need to explicitly disable things in ./configure to get a working build on my x86-64 machines.

What does not work currently or is not implemented:

  1. Detection of CPU for avx2 and avx512 is incomplete so you need to use ./configure --disable-avx2 if the machine does not have avx2 (this is easy to fix though).
  2. Detection of CPU for whether or not to use assembly is incomplete so you need to use --disable-assembly if your CPU does not support the assembly.
  3. Detection of assembly variables for config.m4 is not implemented. Currently it just hardcodes everything including LSYM_PREFIX='L' which gives the right values for macos/clang/arm64 where I can build and use the assembly (my x86_64 machines don't support the broadwell assembly). The checks in acinclude.m4 would need to be translated to meson/python for this to work I think. Also LSYM_PREFIX='L' is used for the assembly in longlong.h on x86_64 which works in my tests.
  4. In principle it is not difficult to add but I haven't tested --with-gc, --with-ntl so those are disabled for now.
  5. The case of --with-blas is trickier because there are multiple BLAS libraries and meson does not yet have a way to ask for a generic BLAS library rather than naming particular libraries like openblas.
  6. It is not yet clear to me how to handle --host. In meson this is done with native files and cross files I think but I'm not completely sure yet. For now you can think of this as just using the default --host triple except that it might include avx, assembly etc as configured. You can also set e.g. CFLAGS ='-march=native' when calling ./configure if building with gcc/clang.
  7. Need to add something that calls config.guess for CPU detection and then implement the logic in configure.ac that translates the output to gcc flags, mparam, asm path, have_avx.

I don't think any of these things is particularly difficult if just translating or reusing the existing code. I'm not sure if there are other significant checks or options missing. It just takes a bit of time to translate and test each of the configurations and check what the existing configure logic is doing.

@albinahlback
Copy link
Collaborator

Btw, this will require users to have Python installed. I know that there was a crossover period between Python 2 and Python 3, and there is incompatibilities between them. Does every modern OS distribute Python 3 instead of Python 2 these days?

@edgarcosta
Copy link
Member

I would say so. I do not know a single system that doesn't come with Python 3.
I think something like GMP 6.2.1 is a harder requirement, e.g., Ubuntu Focal, now only providing security updates and LTS expiring in 2025, only provides GMP 6.2.0, but I wouldn't worry about that.

@albinahlback
Copy link
Collaborator

I would say so. I do not know a single system that doesn't come with Python 3. I think something like GMP 6.2.1 is a harder requirement, e.g., Ubuntu Focal, now only providing security updates and LTS expiring in 2025, only provides GMP 6.2.0, but I wouldn't worry about that.

I mean, Ubuntu 20.04 is about 6 years old software-wise, so I don't think we should worry about that as you say.

@oscarbenjamin
Copy link
Contributor Author

Does Flint have any policy about support for being able to build Flint on older Linux distros?

I would pay more attention to the minimum Python 3 version rather than availability of Python 2. Current minimum Python version for meson is Python 3.7:
https://github.com/mesonbuild/meson/blob/9e3b3db7054c7dedecd14db3e6061ff7e2227faf/setup.py#L8-L10
There is a little table in the README of maximum meson versions for any given Python version:
https://github.com/mesonbuild/meson

To have Python 3.6 as the minimum Python version we need meson 0.61.5 as the minimum meson version. To support Python 3.6 as the oldest Python version we can set that as the oldest meson version:

diff --git a/_meson_build/meson.build b/_meson_build/meson.build
index 96a32fffc..a6d60fb58 100644
--- a/_meson_build/meson.build
+++ b/_meson_build/meson.build
@@ -1,6 +1,7 @@
 project('FLINT', 'c',
   version: '3.2.0-dev',
   license: 'LGPL3+',
+  meson_version: '>= 0.61.5',
   default_options: [
     'buildtype=release',
     'c_std=c11',

Then if you run meson you get warnings about using meson features that are not supported in 0.61.5:

$ ./configure
--------------------------------------------------------------------------------
$ meson setup build.meson
--------------------------------------------------------------------------------
The Meson build system
Version: 1.4.0
Source dir: /home/oscar/current/active/flint
Build dir: /home/oscar/current/active/flint/build.meson
Build type: native build
WARNING: Project targets '>= 0.61.5' but uses feature introduced in '1.1': meson.options file. Use meson_options.txt instead
Project name: FLINT
Project version: 3.2.0-dev
C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
Host machine cpu family: x86_64
Host machine cpu: x86_64
Library m found: YES
Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
Run-time dependency gmp found: YES 6.2.1
Run-time dependency mpfr found: YES 4.1.0
Checking for function "atan2" with dependency -lm: YES 
Checking if "mpz_init links" with dependency gmp: links: YES 
Checking if "mpn_mul_basecase links" with dependency gmp: links: YES 
Checking if "mpn_gcd_11 links" with dependency gmp: links: YES 
Checking if "mpn_div_q links" with dependency gmp: links: YES 
Checking if "mpn_add_n_sub_n links" with dependency gmp: links: YES 
Checking if "mpn_add_nc links" with dependency gmp: links: YES 
Checking if "mpn_addlsh1_n links" with dependency gmp: links: YES 
Checking if "mpn_addlsh1_n_ip1 links" with dependency gmp: links: NO 
Checking if "mpn_addmul_2 links" with dependency gmp: links: YES 
Checking if "mpn_modexact_1_odd links" with dependency gmp: links: YES 
Checking if "mpn_rsh1add_n links" with dependency gmp: links: YES 
Checking if "mpn_rsh1sub_n links" with dependency gmp: links: YES 
Checking if "mpn_sub_nc links" with dependency gmp: links: YES 
Has header "immintrin.h" : YES 
Has header "arm_neon.h" : NO 
detection/meson.build:197: WARNING: Project targets '>= 0.61.5' but uses feature introduced in '1.1.0': feature_option.enable_auto_if().
Message: FFT_SMALL:  enabled
detection/meson.build:234: WARNING: Project targets '>= 0.61.5' but uses feature introduced in '1.1.0': feature_option.enable_auto_if().
Message: ASSEMBLY:  enabled
Header "stdlib.h" has symbol "aligned_alloc" : YES 
Has header "alloca.h" : YES 
Has header "arm_neon.h" : NO (cached)
Has header "dlfcn.h" : YES 
Has header "errno.h" : YES 
Has header "fenv.h" : YES 
Has header "float.h" : YES 
Has header "immintrin.h" : YES (cached)
Has header "inttypes.h" : YES 
Has header "malloc.h" : YES 
Has header "math.h" : YES 
Has header "pthread_np.h" : NO 
Has header "stdarg.h" : YES 
Has header "stdint.h" : YES 
Has header "stdio.h" : YES 
Has header "stdlib.h" : YES 
Has header "string.h" : YES 
Has header "strings.h" : YES 
Has header "sys/param.h" : YES 
Has header "sys/stat.h" : YES 
Has header "sys/types.h" : YES 
Has header "unistd.h" : YES 
Has header "windows.h" : NO 
Configuring config.m4 using configuration
Program m4 found: YES (/usr/bin/m4)
Configuring flint.h using configuration
Configuring flint-config.h using configuration
Configuring config.h using configuration
include/flint/meson.build:45: WARNING: Project targets '>= 0.61.5' but uses feature introduced in '0.64.0': fs.copyfile.
Build targets in project: 180
WARNING: Project specifies a minimum meson_version '>= 0.61.5' but uses features which were added in newer versions:
 * 0.64.0: {'fs.copyfile'}
 * 1.1: {'meson.options file'}
 * 1.1.0: {'feature_option.enable_auto_if()'}
NOTICE: Future-deprecated features used:
 * 0.64.0: {'copy arg in configure_file'}

Found ninja-1.10.1 at /home/oscar/.pyenv/shims/ninja

Some of these meson features can easily be avoided but some might be more awkward.

For now the question then is what minimum Python version to target and whether specific versions of meson are being shipped in old distros. It is easy to install meson with pip if needed but if distros already have meson/ninja then it would be useful to be able to support the versions they supply.

Ubuntu 19.04 is the oldest Ubuntu version to ship with Python 3.7 as the system Python. It looks like it ships with an older version of meson (0.45.1):
https://launchpad.net/meson/+packages
Someone on Ubuntu 19.04 could easily install a newer version of meson with pip but that might be a bit awkward for some Flint users.

@oscarbenjamin
Copy link
Contributor Author

Ubuntu 19.04 is the oldest Ubuntu version to ship with Python 3.7 as the system Python. It looks like it ships with an older version of meson (0.45.1):

These are the warnings if putting 0.45.1 as the minimum meson version:

WARNING: Project specifies a minimum meson_version '>= 0.45.1' but uses features which were added in newer versions:
 * 0.47.0: {'copy arg in configure_file', 'dict'}
 * 0.49.0: {'/ with string arguments', 'configure_file.configuration dictionary'}
 * 0.50.0: {'include_directories kwarg of type string'}
 * 0.53.0: {'module fs'}
 * 0.54.0: {'fs.stem', 'message with more than one argument'}
 * 0.58.0: {'format strings'}
 * 0.59.0: {'feature_option.allowed()', 'feature_option.require()'}
 * 0.64.0: {'fs.copyfile'}
 * 1.1: {'meson.options file'}
 * 1.1.0: {'feature_option.enable_auto_if()'}
NOTICE: Future-deprecated features used:
 * 0.64.0: {'copy arg in configure_file'}

I don't think it would be hard to avoid using all of those features but it would complicate things a little. I haven't actually tested whether everything would work though.

@albinahlback
Copy link
Collaborator

Does Flint have any policy about support for being able to build Flint on older Linux distros?

No, but there are two things that has to be outweighed: backwards-compatibility and maintainability.

We want to be as backwards-compatible as possible, but we also want the process of maintaining to be as easy as possible. Hence, we have to weigh the two different cases and optimize it accordingly. For me, if there is something that makes it much more maintainable which would require a version that is two or less years old, I would vote for using it. I base this off of the latest Ubuntu LTS which at most is two years old.

Less notable, it is also important IMO to be modern to make contributions less of a hassle for newcomers. If we rely on 20 year software, newcomers may have a harder time understanding the source code and, in the process, may avoid to contribute. Of course, this could be said for Autotools as well.

I would pay more attention to the minimum Python 3 version rather than availability of Python 2. Current minimum Python version for meson is Python 3.7: https://github.com/mesonbuild/meson/blob/9e3b3db7054c7dedecd14db3e6061ff7e2227faf/setup.py#L8-L10 There is a little table in the README of maximum meson versions for any given Python version: https://github.com/mesonbuild/meson

That seems fine as 3.7 was released 6 years ago.

detection/meson.build Outdated Show resolved Hide resolved
@dimpase
Copy link
Contributor

dimpase commented Jun 19, 2024

What is exactly the problem with the cmake build, expect lack of documentation? cmake nowadays plays nicely with Python, and projects like pyzmq/0mq are switching to cmake after trying meson.
So, perhaps, fixing whatever is not OK with cmake build here might be easier/more useful.

@oscarbenjamin
Copy link
Contributor Author

What is exactly the problem with the cmake build, expect lack of documentation?

I don't have any problem with cmake except that I don't personally know it. I think that having Flint use either meson or cmake would solve the original problems that motivated writing this.

I think that the meson build here already implements more of the features of the existing autotools build than the cmake build does. The main thing missing both from here and also from the cmake build is running config.guess to get the CPU and using that to set compiler flags etc. Otherwise I think the meson build here is just missing a few features like --with-blas.

An alternative to having this be the standard build system for Flint is that it can be added to the meson patch database. Then it would still be possible for downstream projects to use meson as the build system for Flint if there is a need to do so.

@dimpase
Copy link
Contributor

dimpase commented Jun 19, 2024

maybe @minrk can comment on the choice of cmake for pyzmq

@albinahlback
Copy link
Collaborator

What is exactly the problem with the cmake build, expect lack of documentation? cmake nowadays plays nicely with Python, and projects like pyzmq/0mq are switching to cmake after trying meson. So, perhaps, fixing whatever is not OK with cmake build here might be easier/more useful.

The problem is that we (more specifically myself) do not want to maintain multiple different building systems for a single project. We have CMake, but that is only for Windows, and I will not maintain it for other systems. For instance, CMake does not check some things it would need to check since it assumes a Windows build, and it cannot build the assembly code.

So the solution is to replace all building systems in FLINT with a single building system.

@minrk
Copy link

minrk commented Jun 20, 2024

I think either CMake via scikit-build-core or meson-py is a fine choice, you definitely don't need both, and both are pretty equivalent capability-wise. CMake is much older and more mature, so there's both a lot more baggage and a lot more precedence/documentation, which I think makes it easier to find out how to do things with CMake.

I initially tried to use meson-py, but struggled to implement the pyzmq logic of "maybe bundle libzmq." libzmq having its own cmake build system made it easier to integrate it into pyzmq's own as well (meson can technically try to do this, too). Once I was able to figure out how to do what I wanted in CMake, I think the same solution could have worked in meson, but it was definitely easier for me to iterate and figure it out in CMake. I had no experience with meson and very little experience with CMake when I did this.

One problem I did encounter was that the configure stage of meson-py was incapable of informing the project that it is building a source distribution, which means it goes through the whole configure stage for building a wheel just to build an sdist, and then does it again to build the wheel from the sdist when using standard repo->sdist->wheel workflow. In my case, that actually meant sdist builds actually failed because unused dependencies were missing and I couldn't find a way to ignore them. scikit-build-core has much simpler, more Pythonic, and more controllable sdist behavior.

Meson is a lot more restrictive and I got a fair amount of purity-beats-practicality feeling from the project when looking up how to do things, which I personally found off-putting. Meson is a much nicer, more modern system and language, especially if your build and dependencies are straightforward.

@oscarbenjamin
Copy link
Contributor Author

I think either CMake via scikit-build-core or meson-py is a fine choice

Note that meson-python and meson are not the same thing. Flint is not a Python library and it does not need a Python PEP 517 build backend like meson-python or scikit-build-core. Rather it needs a general purpose build system for non-Python projects like meson or cmake. It is easy to get confused between these because meson-python obviously uses meson and likewise scikit-build-core uses CMake but they are not the same thing.

On the other hand python-flint is a Python library that wraps Flint and uses meson-python. In that situation it is much nicer if Flint can be built with either meson or CMake rather than autotools so that it can be incorporated as a subproject in python-flint's meson build.

I can't speak for CMake or scikit-build-core but if considering the quality of documentation or maturity it is important to distinguish meson from meson-python again. The meson project's first release was in 2016 whereas meson-python's first release was 2021. Lots of non-Python projects use meson and lots of projects that don't use meson can still be built with meson. Anyone using meson-python (rather than meson) to build Python projects is an early adopter at this stage though.

One problem I did encounter was that the configure stage of meson-py was incapable of informing the project that it is building a source distribution

This is not relevant to Flint or meson (rather than meson-python and python-flint). I can see the same when building an sdist for python-flint but I don't consider it to be a significant problem.

The relevant command for Flint would be meson dist which does not yet work with the changes in this PR. I think that the meson.build files would need to be committed to git for that to work.

@minrk
Copy link

minrk commented Jun 20, 2024

Thanks, I don't actually know what flint is, so I jotted down my experience from pyzmq. I think the real deciding factor should be what do maintainers like the look of, and, if you need another reason, follow what neighboring projects in your community seem to be using (e.g. if python-flint uses meson-python, using meson makes perfect sense, especially if there is contributor overlap). I don't have a technical argument in favor of one or the other.

I will say the scope and maturity of docs/examples/Q&A point was about cmake vs meson and not scikit-build-core vs meson-python, both of which are similarly young projects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants