Internal server error after complete installation of ovirt-engine 4.5

[ghost]

Description of the problem

Type of problem?

500 internal server issue

when we try to access oVirt-engine every time we facing 500 error.
oVirt-engine 4.5/Centos-8
please help us to resolve this issue......thanks in advance.

Enter details ...

Help needed from (use @mention to request assistance)

[sandrobonazzola]

@sudbhat88 can you please provide engine version and attach the reported generated by sosreport -o ovirt on the engine host?

WARNING

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

[charnet1019]

Same error
sosreport-ovirt454-811-2023-03-08-qmzixay.tar.zip

[xiaonaiquan]

I have the same error @sandrobonazzola

[sandrobonazzola]

I don't see any relevant error in server log, @mwperina any suggestion?

[Tararashka]

I have the same error

There were errors during installation

` Running scriptlet: leveldb-1.20-1.el8s.x86_64 213/518
Installing : qpid-proton-c-0.35.0-3.el8s.x86_64 214/518
Installing : python3-qpid-proton-0.35.0-3.el8s.x86_64 215/518
Installing : python3-pyngus-2.3.0-4.el8.noarch 216/518
Running scriptlet: openvswitch-selinux-extra-policy-1.0-29.el8s.noarch 217/518
Installing : openvswitch-selinux-extra-policy-1.0-29.el8s.noarch 217/518
Running scriptlet: openvswitch-selinux-extra-policy-1.0-29.el8s.noarch 217/518
Running scriptlet: openvswitch2.15-2.15.0-136.el8s.x86_64 218/518
Installing : openvswitch2.15-2.15.0-136.el8s.x86_64 218/518
Running scriptlet: openvswitch2.15-2.15.0-136.el8s.x86_64 218/518
Running scriptlet: ovirt-openvswitch-2.15-4.el8.noarch 219/518
Failed to get unit file state for ovirt-provider-ovn.service: No such file or directory
Failed to get unit file state for ovn-controller.service: No such file or directory

Installing : ovirt-openvswitch-2.15-4.el8.noarch 219/518
Installing : python3-openvswitch2.15-2.15.0-136.el8s.x86_64 220/518
Installing : ovirt-python-openvswitch-2.15-4.el8.noarch `

the error is repeated with the repetition of the night snapshot "oVirt master snapshot" and with "centos-release-ovirt45"

CentOS Stream release 8

[codeinsidepro]

I am getting "500 - Internal Server Error" on rocky linux 8

[MadmanXML]

same error in centos stream 9;
ovirt version is 4.5.6.
the following error occurred while accessing https:///ovirt-engine/webadmin/?locale=en_US

'''
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.
'''

I have already reinstalled it many times.
additional informations:

1. During dnf install ovirt-engine:

'''
 install   : ovirt-openvswitch-ovn-common-2.17-1.el9.noarch                                                                                                                                                           83/392 
  run script: ovn22.09-central-22.09.0-31.el9s.x86_64                                                                                                                                                                  84/392 
Unit ovn-northd.service could not be found.

  install    : ovn22.09-central-22.09.0-31.el9s.x86_64 
...

run script: ovirt-openvswitch-2.17-1.el9.noarch                                                                                                                                                                     346/392 
Failed to get unit file state for ovirt-provider-ovn.service: No such file or directory
Failed to get unit file state for ovn-controller.service: No such file or directory

 install    : ovirt-openvswitch-2.17-1.el9.noarch  
...
'''

'''
but I run this:

[root@NEGINE ~]# rpm -qa | grep ovn
ovn22.09-22.09.0-31.el9s.x86_64
ovirt-openvswitch-ovn-2.17-1.el9.noarch
novnc-1.3.0-1.el9s.noarch
ovirt-openvswitch-ovn-common-2.17-1.el9.noarch
ovn22.09-central-22.09.0-31.el9s.x86_64
ovirt-openvswitch-ovn-central-2.17-1.el9.noarch
ovirt-provider-ovn-1.2.36-1.el9.noarch
[root@NEGINE ~]# 
[root@NEGINE ~]# systemctl status ovn-northd.service
○ ovn-northd.service - OVN northd management daemon
     Loaded: loaded (/usr/lib/systemd/system/ovn-northd.service; disabled; preset: disabled)
     Active: inactive (dead)
[root@NEGINE ~]# systemctl start ovn-northd.service
[root@NEGINE ~]# systemctl status ovn-northd.service
● ovn-northd.service - OVN northd management daemon
     Loaded: loaded (/usr/lib/systemd/system/ovn-northd.service; disabled; preset: disabled)
     Active: active (exited) since Thu 2024-05-30 12:53:52 CST; 2s ago
    Process: 11291 ExecStartPre=/usr/bin/chown -R ${OVN_USER_ID} ${OVN_DBDIR} (code=exited, status=0/SUCCESS)
    Process: 11292 ExecStart=/usr/share/ovn/scripts/ovn-ctl --ovn-user=${OVN_USER_ID} start_northd $OVN_NORTHD_OPTS (code=exited, status=0/SUCCESS)
   Main PID: 11292 (code=exited, status=0/SUCCESS)
      Tasks: 8 (limit: 48588)
     Memory: 11.5M
        CPU: 109ms
     CGroup: /system.slice/ovn-northd.service
             ├─11318 "ovsdb-server: monitoring pid 11319 (healthy)"
             ├─11319 ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/ovn/ovsdb-server-nb.log --remote=punix:/run/ovn/ovnnb_db.sock --pidfile=/run/ovn/ovnnb_db.pid --unixctl=/run/ovn/ovnnb_db.ctl --user openvs>
             ├─11334 "ovsdb-server: monitoring pid 11335 (healthy)"
             ├─11335 ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/ovn/ovsdb-server-sb.log --remote=punix:/run/ovn/ovnsb_db.sock --pidfile=/run/ovn/ovnsb_db.pid --unixctl=/run/ovn/ovnsb_db.ctl --user openvs>
             ├─11345 "ovn-northd: monitoring pid 11346 (healthy)"
             └─11346 ovn-northd --user openvswitch:openvswitch -vconsole:emer -vsyslog:err -vfile:info --ovnnb-db=unix:/run/ovn/ovnnb_db.sock --ovnsb-db=unix:/run/ovn/ovnsb_db.sock --no-chdir --log-file=/var/log/ovn/ovn>

5月 30 12:53:51 NEGINE.xml.com ovn-ctl[11292]: Starting ovsdb-nb.
5月 30 12:53:51 NEGINE.xml.com ovn-nbctl[11323]: ovs|00001|ovn_dbctl|INFO|Called as ovn-nbctl --no-leader-only --db=unix:/run/ovn/ovnnb_db.sock init
5月 30 12:53:52 NEGINE.xml.com ovn-ctl[11292]: /var/lib/ovn/ovnsb_db.db does not exist ... (warning).
5月 30 12:53:52 NEGINE.xml.com ovn-ctl[11292]: Creating empty database /var/lib/ovn/ovnsb_db.db.
5月 30 12:53:52 NEGINE.xml.com ovsdb-server[11333]: ovs|00001|vlog|INFO|opened log file /var/log/ovn/ovsdb-server-sb.log
5月 30 12:53:52 NEGINE.xml.com ovsdb-server[11335]: ovs|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.17.10
5月 30 12:53:52 NEGINE.xml.com ovn-ctl[11292]: Starting ovsdb-sb.
5月 30 12:53:52 NEGINE.xml.com ovn-sbctl[11339]: ovs|00001|ovn_dbctl|INFO|Called as ovn-sbctl --no-leader-only --db=unix:/run/ovn/ovnsb_db.sock init
5月 30 12:53:52 NEGINE.xml.com ovn-ctl[11292]: Starting ovn-northd.
5月 30 12:53:52 NEGINE.xml.com systemd[1]: Finished OVN northd management daemon.
[root@NEGINE ~]# 
'''

3. The setup process did not have an exception:

'''
[root@ENGINE ~]# engine-setup 
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf
          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20240530130203-2w5f99.log
          Version: otopi-1.10.4 (otopi-1.10.4-1.el9)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup (late)
[ INFO  ] Stage: Environment customization
         
          --== PRODUCT OPTIONS ==--
         
          Configure Cinderlib integration (Currently in tech preview) (Yes, No) [No]:
          Configure Engine on this host (Yes, No) [Yes]:
         
          Configuring ovirt-provider-ovn also sets the Default cluster's default network provider to ovirt-provider-ovn.
          Non-Default clusters may be configured with an OVN after installation.
          Configure ovirt-provider-ovn (Yes, No) [Yes]:
          Configure WebSocket Proxy on this host (Yes, No) [Yes]:
         
          * Please note * : Data Warehouse is required for the engine.
          If you choose to not configure it on this host, you have to configure
          it on a remote host, and then configure the engine on this host so
          that it can access the database of the remote Data Warehouse host.
          Configure Data Warehouse on this host (Yes, No) [Yes]:
         
          * Please note * : Keycloak is now deprecating AAA/JDBC authentication module.
          It is highly recommended to install Keycloak based authentication.
          Configure Keycloak on this host (Yes, No) [Yes]:
          Configure VM Console Proxy on this host (Yes, No) [Yes]:
          Configure Grafana on this host (Yes, No) [Yes]:
         
          --== PACKAGES ==--
         
[ INFO  ] Checking for product updates...
[ INFO  ] No product updates found
         
          --== NETWORK CONFIGURATION ==--
         
          Host fully qualified DNS name of this server [Engine.xml.com]:
         
          Setup can automatically configure the firewall on this system.
          Note: automatic configuration of the firewall may overwrite current settings.
          Do you want Setup to configure the firewall? (Yes, No) [Yes]:
[ INFO  ] firewalld will be configured as firewall manager.
         
          --== DATABASE CONFIGURATION ==--
         
          Where is the DWH database located? (Local, Remote) [Local]:
         
          Setup can configure the local postgresql server automatically for the DWH to run. This may conflict with existing applications.
          Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? (Automatic, Manual) [Automatic]:
          Where is the Keycloak database located? (Local, Remote) [Local]:
         
          Setup can configure the local postgresql server automatically for the Keycloak to run. This may conflict with existing applications.
          Would you like Setup to automatically configure postgresql and create Keycloak database, or prefer to perform that manually? (Automatic, Manual) [Automatic]:
          Where is the Engine database located? (Local, Remote) [Local]:
         
          Setup can configure the local postgresql server automatically for the engine to run. This may conflict with existing applications.
          Would you like Setup to automatically configure postgresql and create Engine database, or prefer to perform that manually? (Automatic, Manual) [Automatic]:
         
          --== OVIRT ENGINE CONFIGURATION ==--
         
          Engine admin password:
          Confirm engine admin password:
[WARNING] Password is weak: 密码未通过字典检查 - 它基于一个字典中的词
          Use weak password? (Yes, No) [No]:yes
          Application mode (Virt, Gluster, Both) [Both]:
          Use Engine admin password as initial keycloak admin password (Yes, No) [Yes]:
         
          --== STORAGE CONFIGURATION ==--
         
          Default SAN wipe after delete (Yes, No) [No]:
         
          --== PKI CONFIGURATION ==--
         
          Organization name for certificate [xml.com]:
         
          --== APACHE CONFIGURATION ==--
         
          Setup can configure the default page of the web server to present the application home page. This may conflict with existing applications.
          Do you wish to set the application as the default page of the web server? (Yes, No) [Yes]:
         
          Setup can configure apache to use SSL using a certificate issued from the internal CA.
          Do you wish Setup to configure that, or prefer to perform that manually? (Automatic, Manual) [Automatic]:
         
          --== SYSTEM CONFIGURATION ==--
         
         
          --== MISC CONFIGURATION ==--
         
          Please choose Data Warehouse sampling scale:
          (1) Basic
          (2) Full
          (1, 2)[1]:
          Use Engine admin password as initial Grafana admin password (Yes, No) [Yes]:
         
          --== END OF CONFIGURATION ==--
         
[ INFO  ] Stage: Setup validation
[WARNING] Less than 16384MB of memory is available
         
          --== CONFIGURATION PREVIEW ==--
         
          Application mode                        : both
          Default SAN wipe after delete           : False
          Host FQDN                               : Engine.xml.com
          Firewall manager                        : firewalld
          Update Firewall                         : True
          Set up Cinderlib integration            : False
          Configure local Engine database         : True
          Set application as default page         : True
          Configure Apache SSL                    : True
          Keycloak installation                   : True
          Engine database host                    : localhost
          Engine database port                    : 5432
          Engine database secured connection      : False
          Engine database host name validation    : False
          Engine database name                    : engine
          Engine database user name               : engine
          Engine installation                     : True
          PKI organization                        : xml.com
          Set up ovirt-provider-ovn               : True
          DWH installation                        : True
          DWH database host                       : localhost
          DWH database port                       : 5432
          DWH database secured connection         : False
          DWH database host name validation       : False
          DWH database name                       : ovirt_engine_history
          Configure local DWH database            : True
          Grafana integration                     : True
          Grafana database user name              : ovirt_engine_history_grafana
          Keycloak database host                  : localhost
          Keycloak database port                  : 5432
          Keycloak database secured connection    : False
          Keycloak database host name validation  : False
          Keycloak database name                  : ovirt_engine_keycloak
          Keycloak database user name             : ovirt_engine_keycloak
          Configure local Keycloak database       : True
          Configure VMConsole Proxy               : True
          Configure WebSocket Proxy               : True
         
          Please confirm installation settings (OK, Cancel) [OK]:
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stopping engine service
[ INFO  ] Stopping ovirt-fence-kdump-listener service
[ INFO  ] Stopping dwh service
[ INFO  ] Stopping vmconsole-proxy service
[ INFO  ] Stopping websocket-proxy service
[ INFO  ] Stage: Misc configuration (early)
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Upgrading CA
[ INFO  ] Initializing PostgreSQL
[ INFO  ] Creating PostgreSQL 'engine' database
[ INFO  ] Configuring PostgreSQL

Broadcast message from [email protected] (Thu 2024-05-30 13:04:38 CST):

sssd_kcm[2682]: Could not open file [/var/log/sssd/sssd_kcm.log]. Error: [2][No such file or directory]

[ INFO  ] Creating PostgreSQL 'ovirt_engine_history' database
[ INFO  ] Configuring PostgreSQL

Message from syslogd@ENGINE at May 30 13:04:38 ...
 sssd_kcm[2682]:Could not open file [/var/log/sssd/sssd_kcm.log]. Error: [2][No such file or directory]
[ INFO  ] Creating PostgreSQL 'ovirt_engine_keycloak' database
[ INFO  ] Configuring PostgreSQL
[ INFO  ] Creating CA: /etc/pki/ovirt-engine/ca.pem
[ INFO  ] Creating CA: /etc/pki/ovirt-engine/qemu-ca.pem
[ INFO  ] Creating a user for Grafana
[ INFO  ] Allowing ovirt_engine_history_grafana to read data on ovirt_engine_history
[ INFO  ] Setting up ovirt-vmconsole proxy helper PKI artifacts
[ INFO  ] Setting up ovirt-vmconsole SSH PKI artifacts
[ INFO  ] Configuring WebSocket Proxy
[ INFO  ] Creating/refreshing Engine database schema
[ INFO  ] Creating/refreshing DWH database schema
[ INFO  ] Updating OVN SSL configuration
[ INFO  ] Updating OVN timeout configuration
[ INFO  ] Creating/refreshing Engine 'internal' domain database schema
[ INFO  ] Creating default mac pool range
[ INFO  ] Adding default OVN provider to database
[ INFO  ] Adding OVN provider secret to database
[ INFO  ] Setting a password for internal user admin
[ INFO  ] Creating initial Keycloak admin user
[ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
         
          --== SUMMARY ==--
         
[ INFO  ] No need to restart fapolicyd because it is not running.
[ INFO  ] Starting dwh service
[ INFO  ] Starting Grafana service
[ INFO  ] Restarting ovirt-vmconsole proxy service
          To login to oVirt using Keycloak SSO, enter 'admin@ovirt' as username and the password provided during Setup
          To login to Keycloak Administration Console enter 'admin' as username and the password provided during Setup
          Web access for Keycloak Administration Console is enabled at:
              https://Engine.xml.com/ovirt-engine-auth/admin
          Web access is enabled at:
              http://Engine.xml.com:80/ovirt-engine
              https://Engine.xml.com:443/ovirt-engine
          Internal CA fingerprint: SHA256: 8B:83:DF:D7:6D:56:9A:6F:03:CF:D5:72:8A:F8:16:19:16:19:B5:A4:95:31:D5:99:71:07:88:1F:C9:57:81:92
          SSH fingerprint: SHA256:+m07tbMGTZ58RD7h957OzxYEANACZIzvvBQrmLuLS2s
[ INFO  ] Starting engine service
[WARNING] Less than 16384MB of memory is available
          Web access for grafana is enabled at:
              https://Engine.xml.com/ovirt-engine-grafana/
          Please run the following command on the engine machine Engine.xml.com, for SSO to work:
          systemctl restart ovirt-engine
         
          --== END OF SUMMARY ==--
         
[ INFO  ] Restarting httpd
[ INFO  ] Start with setting up Keycloak for Ovirt Engine
[ INFO  ] Done with setting up Keycloak for Ovirt Engine
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20240530130203-2w5f99.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20240530130652-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Execution of setup completed successfully
[root@ENGINE ~]# 
'''

I really want to know what the problem is and how do we fix it?