Compiling / running on Fedora based system (RHEL 8), specifically Rocky 8.6

[mwmalopo]

I apologize if this has been asked before, I did some checking and didn't find anything. I am trying to compile and use OpenAvnu (with the igb_avb driver) on a Fedora based system (RHEL 8 / Rocky 8.6). OpenAvnu compiles fine, but there are issues compiling the igb_avb kernel module that don't completely make sense. Note that I have used this in RHEL 7 / CentOS 7 successfully.

Output from uname (on Rocky 8.6 system):

Linux hostname 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

First compilation error:

[user@hostname OpenAvnu-july]$ make -C lib/igb_avb/ kmod
make: Entering directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb'
make  subdir=kmod  -C kmod 
make[1]: Entering directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod'
make -C /lib/modules/4.18.0-372.9.1.el8.x86_64/build SUBDIRS=/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod modules
make[2]: Entering directory '/usr/src/kernels/4.18.0-372.9.1.el8.x86_64'
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.o
In file included from /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb.h:61,
                 from /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:69:
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/kcompat.h:2579:10: fatal error: linux/pci-aspm.h: No such file or directory
 #include <linux/pci-aspm.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [scripts/Makefile.build:316: /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.o] Error 1
make[2]: *** [Makefile:1577: _module_/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod] Error 2
make[2]: Leaving directory '/usr/src/kernels/4.18.0-372.9.1.el8.x86_64'
make[1]: *** [Makefile:262: default] Error 2
make[1]: Leaving directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod'
make: *** [Makefile:13: kmod] Error 2
make: Leaving directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb'

I have tried various different versions (commits) and even though my kernel is 4.18 and does not seem to have linux/pci-aspm.h the check is looking for less than a 5.x kernel (assuming this is mostly developed against Ubuntu / Debian flavored systems). So if I manually remove / comment that line (easy enough), this leads to the next issue:

[user@hostname OpenAvnu-july]$ make -C lib/igb_avb/ kmod
make: Entering directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb'
make  subdir=kmod  -C kmod 
make[1]: Entering directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod'
make -C /lib/modules/4.18.0-372.9.1.el8.x86_64/build SUBDIRS=/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod modules
make[2]: Entering directory '/usr/src/kernels/4.18.0-372.9.1.el8.x86_64'
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.o
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:239:13: error: initialization of ‘vm_fault_t (*)(struct vm_fault *)’ {aka ‘unsigned int (*)(struct vm_fault *)’} from incompatible pointer type ‘int (*)(struct vm_fault *)’ [-Werror=incompatible-pointer-types]
   .fault  = igb_vm_fault
             ^~~~~~~~~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:239:13: note: (near initialization for ‘igb_mmap_ops.fault’)
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:2360:22: error: initialization of ‘u16 (*)(struct net_device *, struct sk_buff *, struct net_device *, u16 (*)(struct net_device *, struct sk_buff *, struct net_device *))’ {aka ‘short unsigned int (*)(struct net_device *, struct sk_buff *, struct net_device *, short unsigned int (*)(struct net_device *, struct sk_buff *, struct net_device *))’} from incompatible pointer type ‘u16 (*)(struct net_device *, struct sk_buff *, void *, u16 (*)(struct net_device *, struct sk_buff *, struct net_device *))’ {aka ‘short unsigned int (*)(struct net_device *, struct sk_buff *, void *, short unsigned int (*)(struct net_device *, struct sk_buff *, struct net_device *))’} [-Werror=incompatible-pointer-types]
  .ndo_select_queue = igb_select_queue,
                      ^~~~~~~~~~~~~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:2360:22: note: (near initialization for ‘igb_netdev_ops.ndo_select_queue’)
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:2366:21: error: initialization of ‘void (*)(struct net_device *, unsigned int)’ from incompatible pointer type ‘void (*)(struct net_device *)’ [-Werror=incompatible-pointer-types]
  .ndo_tx_timeout  = igb_tx_timeout,
                     ^~~~~~~~~~~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:2366:21: note: (near initialization for ‘igb_netdev_ops.<anonymous>.ndo_tx_timeout’)
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:2394:18: error: initialization of ‘int (*)(struct ndmsg *, struct nlattr **, struct net_device *, const unsigned char *, u16,  u16,  struct netlink_ext_ack *)’ {aka ‘int (*)(struct ndmsg *, struct nlattr **, struct net_device *, const unsigned char *, short unsigned int,  short unsigned int,  struct netlink_ext_ack *)’} from incompatible pointer type ‘int (*)(struct ndmsg *, struct nlattr **, struct net_device *, const unsigned char *, u16,  u16)’ {aka ‘int (*)(struct ndmsg *, struct nlattr **, struct net_device *, const unsigned char *, short unsigned int,  short unsigned int)’} [-Werror=incompatible-pointer-types]
  .ndo_fdb_add  = igb_ndo_fdb_add,
                  ^~~~~~~~~~~~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:2394:18: note: (near initialization for ‘igb_netdev_ops.<anonymous>.ndo_fdb_add’)
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:2400:24: error: initialization of ‘int (*)(struct net_device *, struct nlmsghdr *, u16,  struct netlink_ext_ack *)’ {aka ‘int (*)(struct net_device *, struct nlmsghdr *, short unsigned int,  struct netlink_ext_ack *)’} from incompatible pointer type ‘int (*)(struct net_device *, struct nlmsghdr *, u16)’ {aka ‘int (*)(struct net_device *, struct nlmsghdr *, short unsigned int)’} [-Werror=incompatible-pointer-types]
  .ndo_bridge_setlink = igb_ndo_bridge_setlink,
                        ^~~~~~~~~~~~~~~~~~~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:2400:24: note: (near initialization for ‘igb_netdev_ops.<anonymous>.ndo_bridge_setlink’)
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c: In function ‘igb_io_error_detected’:
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.c:9579:3: error: implicit declaration of function ‘pci_cleanup_aer_uncorrect_error_status’ [-Werror=implicit-function-declaration]
   pci_cleanup_aer_uncorrect_error_status(pdev);
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
make[3]: *** [scripts/Makefile.build:316: /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.o] Error 1
make[2]: *** [Makefile:1577: _module_/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod] Error 2
make[2]: Leaving directory '/usr/src/kernels/4.18.0-372.9.1.el8.x86_64'
make[1]: *** [Makefile:262: default] Error 2
make[1]: Leaving directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod'
make: *** [Makefile:13: kmod] Error 2
make: Leaving directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb'

That fails due to the -Wall flag to gcc, which can obviously be overrode by adding -Wno-incompatible-pointer-types and -Wno-implicit-function-declaration. Doing so leads to the next issue:

[user@hostname OpenAvnu-july]$ make -C lib/igb_avb/ kmod
make: Entering directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb'
make  subdir=kmod  -C kmod 
make[1]: Entering directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod'
make -C /lib/modules/4.18.0-372.9.1.el8.x86_64/build SUBDIRS=/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod modules
make[2]: Entering directory '/usr/src/kernels/4.18.0-372.9.1.el8.x86_64'
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_main.o
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_82575.o
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_i210.o
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_mac.o
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_nvm.o
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_phy.o
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_phy.c: In function ‘e1000_read_phy_reg_mphy’:
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_phy.c:3294:2: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
  if (locked)
  ^~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_phy.c:3296:3: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
   if (!ready)
   ^~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_phy.c: In function ‘e1000_write_phy_reg_mphy’:
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_phy.c:3359:2: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
  if (locked)
  ^~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_phy.c:3361:3: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
   if (!ready)
   ^~
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/e1000_manage.o
  CC [M]  /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_param.o
In file included from ./include/linux/kernel.h:14,
                 from ./include/linux/list.h:9,
                 from ./include/linux/timer.h:5,
                 from ./include/linux/netdevice.h:28,
                 from /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_param.c:26:
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_param.c: In function ‘igb_check_options’:
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/kcompat.h:273:27: error: ‘smp_num_cpus’ undeclared (first use in this function); did you mean ‘setup_max_cpus’?
 #define num_online_cpus() smp_num_cpus
                           ^~~~~~~~~~~~
./include/linux/minmax.h:18:39: note: in definition of macro ‘__typecheck’
  (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
                                       ^
./include/linux/minmax.h:42:24: note: in expansion of macro ‘__safe_cmp’
  __builtin_choose_expr(__safe_cmp(x, y), \
                        ^~~~~~~~~~
./include/linux/minmax.h:110:27: note: in expansion of macro ‘__careful_cmp’
 #define min_t(type, x, y) __careful_cmp((type)(x), (type)(y), <)
                           ^~~~~~~~~~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_param.c:647:27: note: in expansion of macro ‘min_t’
     adapter->rss_queues = min_t(u32, opt.arg.r.max,
                           ^~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_param.c:648:12: note: in expansion of macro ‘num_online_cpus’
            num_online_cpus());
            ^~~~~~~~~~~~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/kcompat.h:273:27: note: each undeclared identifier is reported only once for each function it appears in
 #define num_online_cpus() smp_num_cpus
                           ^~~~~~~~~~~~
./include/linux/minmax.h:18:39: note: in definition of macro ‘__typecheck’
  (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
                                       ^
./include/linux/minmax.h:42:24: note: in expansion of macro ‘__safe_cmp’
  __builtin_choose_expr(__safe_cmp(x, y), \
                        ^~~~~~~~~~
./include/linux/minmax.h:110:27: note: in expansion of macro ‘__careful_cmp’
 #define min_t(type, x, y) __careful_cmp((type)(x), (type)(y), <)
                           ^~~~~~~~~~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_param.c:647:27: note: in expansion of macro ‘min_t’
     adapter->rss_queues = min_t(u32, opt.arg.r.max,
                           ^~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_param.c:648:12: note: in expansion of macro ‘num_online_cpus’
            num_online_cpus());
            ^~~~~~~~~~~~~~~
./include/linux/minmax.h:42:2: error: first argument to ‘__builtin_choose_expr’ not a constant
  __builtin_choose_expr(__safe_cmp(x, y), \
  ^~~~~~~~~~~~~~~~~~~~~
./include/linux/minmax.h:110:27: note: in expansion of macro ‘__careful_cmp’
 #define min_t(type, x, y) __careful_cmp((type)(x), (type)(y), <)
                           ^~~~~~~~~~~~~
/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_param.c:647:27: note: in expansion of macro ‘min_t’
     adapter->rss_queues = min_t(u32, opt.arg.r.max,
                           ^~~~~
make[3]: *** [scripts/Makefile.build:316: /home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod/igb_param.o] Error 1
make[2]: *** [Makefile:1577: _module_/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod] Error 2
make[2]: Leaving directory '/usr/src/kernels/4.18.0-372.9.1.el8.x86_64'
make[1]: *** [Makefile:262: default] Error 2
make[1]: Leaving directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb/kmod'
make: *** [Makefile:13: kmod] Error 2
make: Leaving directory '/home/user/Tools/openavb/OpenAvnu-july/lib/igb_avb'

At this point it becomes pretty obvious that this doesn't compile without some source changes / assistance. I have done that and can actually get it to compile (e.g., filling in a value for smp_num_cpus, adding NULL to a function call that has an extra parameter now, removing a declared but not defined function).

Once doing that I can unload the igb kernel module and load the igb_avb one but once there is any connection or negotation (at the Ethernet level, not AVB traffic, just normal networking) there is a kernel panic. I can post the kernel panic, but it seems that the first question is to get the complication to work (the right way, not my hacks).

Has anyone successfully compiled and used the igb_avb driver on Fedora / RedHat (RHEL) / Rocky / CentOS 8 systems?

Thank you in advance.