Squashed commit of the following:

commit 4b1576f
Merge: 57a303e 8b0977a
Author: jait <[email protected]>
Date:   Wed Sep 25 22:59:20 2024 +0530

    Merge branch 'fedora-copr:main' into Webhook-History-UI

commit 8b0977a
Author: Pavel Raiskup <[email protected]>
Date:   Tue Sep 24 21:09:24 2024 +0200

    rpmbuild: unblock testsuite

commit 5d77d36
Author: Jiri Kyjovsky <[email protected]>
Date:   Tue Sep 17 14:35:28 2024 +0200

    rpmbuild: specify snippets to mock config via copr-rpmbuild config file

    This allows us to specify tpm fs size to rpmbuild in order to be able to
    automatically generate its size for performance builders.

    See fedora-copr#3268

commit 57a303e
Author: Jait Jacob <[email protected]>
Date:   Wed Sep 25 13:26:34 2024 +0530

    remove unused import & revert localized_time filter method

commit 658230d
Author: Pavel Raiskup <[email protected]>
Date:   Mon Sep 23 15:20:09 2024 +0200

    backend: unknown resalloc tickets helper cleanup

    If no tickets are taken (which often happens in the staging
    environment), this script encountered corner case issues.

commit d6a3472
Author: Miroslav Suchý <[email protected]>
Date:   Mon Sep 23 22:12:43 2024 +0200

    rpmbuild: do not require rpkg,pyp2rpm,pyp2spec,gem2rpm and fedora-review on rhel

    Resolves: RHBZ#2313878

commit 6ae7c6c
Author: Miroslav Suchý <[email protected]>
Date:   Mon Sep 23 22:04:11 2024 +0200

    rpmbuild: do not require qemu-user-static on rhel

    Resolves: RHBZ#2313879

commit 1559b85
Author: Aurélien Bompard <[email protected]>
Date:   Mon Jul 29 10:54:02 2024 +0200

    Use `super()` without argument to make pylint happy

    Signed-off-by: Aurélien Bompard <[email protected]>

commit f3b1643
Author: Aurélien Bompard <[email protected]>
Date:   Mon Jul 29 10:40:13 2024 +0200

    Message schemas: set chroot message severity to DEBUG

    Signed-off-by: Aurélien Bompard <[email protected]>

commit c192726
Author: Aurélien Bompard <[email protected]>
Date:   Mon Jul 29 10:38:34 2024 +0200

    Message schemas: one-line descriptions should be the summary

    Signed-off-by: Aurélien Bompard <[email protected]>

commit 609d369
Author: Pavel Raiskup <[email protected]>
Date:   Fri Sep 20 14:14:31 2024 +0200

    frontend: fix the 500 for racy creation attempts

    This is TOCTOU issue.  The other checks for duplications (on so many
    places) seem kinda redundant because nothing but try/except for commit()
    may catch these concurrency problems.

    Fixes: fedora-copr#3372

commit 446dcb3
Author: Jiri Kyjovsky <[email protected]>
Date:   Wed Sep 18 16:49:52 2024 +0200

    beaker: use podman for testing inside container if installed

commit 030740a
Author: Jait Jacob <[email protected]>
Date:   Sun Sep 22 23:17:17 2024 +0530

    remove unaccessed import

commit cf1cb81
Author: Jait Jacob <[email protected]>
Date:   Fri Sep 20 02:36:09 2024 +0530

    webhook_history stores UNIX timestamps instead of DateTime

commit 6aeb686
Author: Jiri Kyjovsky <[email protected]>
Date:   Mon Sep 16 10:15:37 2024 +0200

    docker: set hard ulimits for docker container

    Because of bug in python3-daemon [1] we need to set ulimits inside
    docker container, otherwise backend and dist-git ooms.

    [1] - https://bugzilla.redhat.com/show_bug.cgi?id=2307635

commit 6aa1d5e
Author: Jait Jacob <[email protected]>
Date:   Sun Sep 15 19:55:11 2024 +0530

    resolve pylint warnings

commit a2adf81
Author: Jait Jacob <[email protected]>
Date:   Sun Sep 15 19:44:07 2024 +0530

    remove unused code

commit 14faa50
Author: Jakub Kadlcik <[email protected]>
Date:   Thu Sep 5 11:10:15 2024 +0200

    frontend, python, cli: allow admins to set storage for new projects

    See fedora-copr#2533

    This will be useful for beaker tests where we can now add basic tests
    for every supported storage.

commit ad36b8b
Author: Jakub Kadlcik <[email protected]>
Date:   Thu Sep 5 10:36:46 2024 +0200

    frontend: make the default storage for new projects configurable

    See fedora-copr#2533

commit 286a913
Author: Jakub Kadlcik <[email protected]>
Date:   Tue Sep 10 08:36:23 2024 +0200

    backend: add a timeout for waiting until a Pulp task finishes

commit 23a2fa3
Author: Jakub Kadlcik <[email protected]>
Date:   Mon Jul 22 11:23:53 2024 +0200

    backend: actions don't call uses_devel_repo function anymore

commit c5166a1
Author: Jakub Kadlcik <[email protected]>
Date:   Thu Sep 5 09:43:09 2024 +0200

    backend, frontend: implement project and build deletion in Pulp

    Fix fedora-copr#3318
    Fix fedora-copr#3319

commit 19eff0c
Author: Jakub Kadlcik <[email protected]>
Date:   Sun Sep 8 12:03:15 2024 +0200

    copr: wait until Pulp publication is finished

commit ff5288d
Author: Miroslav Suchý <[email protected]>
Date:   Wed Sep 11 08:19:41 2024 +0200

    common: cleanup - remove six dependency

commit 6c71993
Author: Jait Jacob <[email protected]>
Date:   Wed Sep 11 16:21:34 2024 +0530

    fix pylint complaint

commit 649fe51
Author: Jait Jacob <[email protected]>
Date:   Wed Sep 11 16:15:14 2024 +0530

    optimize db calls, remove client side js, resolve review comments

commit 31880b4
Author: Jait Jacob <[email protected]>
Date:   Sun Sep 8 22:32:14 2024 +0530

    cleanup

commit 593a13f
Author: Jait Jacob <[email protected]>
Date:   Sun Sep 8 22:17:58 2024 +0530

    add pagination

commit 99e142e
Merge: 11d7fb1 2bdec45
Author: jait <[email protected]>
Date:   Sun Sep 8 15:03:36 2024 +0530

    Merge branch 'fedora-copr:main' into Webhook-History-UI

commit 11d7fb1
Author: Jait Jacob <[email protected]>
Date:   Sun Sep 8 15:03:29 2024 +0530

    return webhook history in descending order & without any duplicates

commit b7f2b34
Author: Jait Jacob <[email protected]>
Date:   Sat Sep 7 00:22:06 2024 +0530

    frontend: show webhook history table under Setting->Integration

backend/copr_backend/actions.py

@@ -20,14 +20,13 @@
 from copr_common.worker_manager import WorkerManager
 
 from copr_backend.worker_manager import BackendQueueTask
-from copr_backend.storage import storage_for_enum
+from copr_backend.storage import storage_for_enum, BackendStorage
 
 from .sign import create_user_keys, CoprKeygenRequestError
 from .exceptions import CreateRepoError, CoprSignError, FrontendClientException
 from .helpers import (get_redis_logger, silent_remove, ensure_dir_exists,
                       get_chroot_arch, format_filename,
-                      uses_devel_repo, call_copr_repo, build_chroot_log_name,
-                      copy2_but_hardlink_rpms)
+                      call_copr_repo, copy2_but_hardlink_rpms)
 from .sign import sign_rpms_in_dir, unsign_rpms_in_dir, get_pubkey
 
 
@@ -64,7 +63,6 @@ def get_action_class(cls, action):
             ActionTypeEnum("rawhide_to_release"): RawhideToRelease,
             ActionTypeEnum("fork"): Fork,
             ActionTypeEnum("build_module"): BuildModule,
-            ActionTypeEnum("delete"): Delete,
             ActionTypeEnum("remove_dirs"): RemoveDirs,
         }.get(action_type, None)
 
@@ -102,8 +100,15 @@ def __init__(self, opts, action, log=None):
             project = self.ext_data.get("projectname")
             devel = self.ext_data.get("devel")
             appstream = self.ext_data.get("appstream")
-            self.storage = storage_for_enum(enum, owner, project, appstream,
-                                            devel, self.opts, self.log)
+            args = [owner, project, appstream, devel, self.opts, self.log]
+            self.storage = storage_for_enum(enum, *args)
+
+            # Even though we already have `self.storage` which uses an
+            # appropriate storage for the project (e.g. Pulp), the project
+            # still has some data on backend (logs, srpm-builds chroot, etc).
+            # Many actions need to be performed on `self.storage` and
+            # `self.backend_storage` at the same time
+            self.backend_storage = storage_for_enum(StorageEnum.backend, *args)
 
     def __str__(self):
         return "<{}(Action): {}>".format(self.__class__.__name__, self.data)
@@ -221,72 +226,24 @@ def run(self):
         return result
 
 
-class Delete(Action):
-    """
-    Abstract class for all other Delete* classes.
-    """
-    # pylint: disable=abstract-method
-    def _handle_delete_builds(self, ownername, projectname, project_dirname,
-                              chroot_builddirs, build_ids, appstream):
-        """ call /bin/copr-repo --delete """
-        devel = uses_devel_repo(self.front_url, ownername, projectname)
-        result = BackendResultEnum("success")
-        for chroot, subdirs in chroot_builddirs.items():
-            chroot_path = os.path.join(self.destdir, ownername, project_dirname,
-                                       chroot)
-            if not os.path.exists(chroot_path):
-                self.log.error("%s chroot path doesn't exist", chroot_path)
-                result = BackendResultEnum("failure")
-                continue
-
-            self.log.info("Deleting subdirs [%s] in %s",
-                          ", ".join(subdirs), chroot_path)
-
-            # Run createrepo first and then remove the files (to avoid old
-            # repodata temporarily pointing at non-existing files)!
-            if chroot != "srpm-builds":
-                # In srpm-builds we don't create repodata at all
-                if not call_copr_repo(chroot_path, delete=subdirs, devel=devel, appstream=appstream,
-                                      logger=self.log):
-                    result = BackendResultEnum("failure")
-
-            for build_id in build_ids or []:
-                log_paths = [
-                    os.path.join(chroot_path, build_chroot_log_name(build_id)),
-                    # we used to create those before
-                    os.path.join(chroot_path, 'build-{}.rsync.log'.format(build_id)),
-                    os.path.join(chroot_path, 'build-{}.log'.format(build_id))]
-                for log_path in log_paths:
-                    try:
-                        os.unlink(log_path)
-                    except OSError:
-                        self.log.debug("can't remove %s", log_path)
-        return result
-
-
-class DeleteProject(Delete):
+class DeleteProject(Action):
     def run(self):
         self.log.debug("Action delete copr")
-        result = BackendResultEnum("success")
-
-        ext_data = json.loads(self.data["data"])
-        ownername = ext_data["ownername"]
-        project_dirnames = ext_data["project_dirnames"]
+        project_dirnames = self.ext_data["project_dirnames"]
 
-        if not ownername:
+        if not self.storage.owner:
             self.log.error("Received empty ownername!")
-            result = BackendResultEnum("failure")
-            return result
+            return BackendResultEnum("failure")
 
         for dirname in project_dirnames:
             if not dirname:
                 self.log.warning("Received empty dirname!")
                 continue
-            path = os.path.join(self.destdir, ownername, dirname)
-            if os.path.exists(path):
-                self.log.info("Removing copr dir %s", path)
-                shutil.rmtree(path)
-        return result
+            self.storage.delete_project(dirname)
+            if not isinstance(self.storage, BackendStorage):
+                self.backend_storage.delete_project(dirname)
+
+        return BackendResultEnum("success")
 
 
 class CompsUpdate(Action):
@@ -322,7 +279,7 @@ def run(self):
         return result
 
 
-class DeleteMultipleBuilds(Delete):
+class DeleteMultipleBuilds(Action):
     def run(self):
         self.log.debug("Action delete multiple builds.")
 
@@ -334,25 +291,24 @@ def run(self):
         #     srpm-builds: [00849545, 00849546]
         #     fedora-30-x86_64: [00849545-example, 00849545-foo]
         #   [...]
-        ext_data = json.loads(self.data["data"])
 
-        ownername = ext_data["ownername"]
-        projectname = ext_data["projectname"]
-        project_dirnames = ext_data["project_dirnames"]
-        build_ids = ext_data["build_ids"]
-        appstream = ext_data["appstream"]
+        project_dirnames = self.ext_data["project_dirnames"]
+        build_ids = self.ext_data["build_ids"]
 
         result = BackendResultEnum("success")
         for project_dirname, chroot_builddirs in project_dirnames.items():
-            if BackendResultEnum("failure") == \
-               self._handle_delete_builds(ownername, projectname,
-                                          project_dirname, chroot_builddirs,
-                                          build_ids, appstream):
+            args = [project_dirname, chroot_builddirs, build_ids]
+            success = self.storage.delete_builds(*args)
+
+            if not isinstance(self.storage, BackendStorage):
+                success = self.backend_storage.delete_builds(*args) and success
+
+            if not success:
                 result = BackendResultEnum("failure")
         return result
 
 
-class DeleteBuild(Delete):
+class DeleteBuild(Action):
     def run(self):
         self.log.info("Action delete build.")
 
@@ -363,25 +319,31 @@ def run(self):
         # chroot_builddirs:
         #   srpm-builds: [00849545]
         #   fedora-30-x86_64: [00849545-example]
-        ext_data = json.loads(self.data["data"])
 
-        try:
-            ownername = ext_data["ownername"]
-            build_ids = [self.data['object_id']]
-            projectname = ext_data["projectname"]
-            project_dirname = ext_data["project_dirname"]
-            chroot_builddirs = ext_data["chroot_builddirs"]
-            appstream = ext_data["appstream"]
-        except KeyError:
+        valid = "object_id" in self.data
+        keys = {"ownername", "projectname", "project_dirname",
+                "chroot_builddirs", "appstream"}
+        for key in keys:
+            if key not in self.ext_data:
+                valid = False
+                break
+
+        if not valid:
             self.log.exception("Invalid action data")
             return BackendResultEnum("failure")
 
-        return self._handle_delete_builds(ownername, projectname,
-                                          project_dirname, chroot_builddirs,
-                                          build_ids, appstream)
+        args = [
+            self.ext_data["project_dirname"],
+            self.ext_data["chroot_builddirs"],
+            [self.data['object_id']],
+        ]
+        success = self.storage.delete_builds(*args)
+        if not isinstance(self.storage, BackendStorage):
+            success = self.backend_storage.delete_builds(*args) and success
+        return BackendResultEnum("success" if success else "failure")
 
 
-class DeleteChroot(Delete):
+class DeleteChroot(Action):
     def run(self):
         self.log.info("Action delete project chroot.")
         chroot = self.ext_data["chrootname"]

backend/copr_backend/pulp.py

@@ -3,6 +3,7 @@
 """
 
 import os
+import time
 import tomllib
 from urllib.parse import urlencode
 import requests
@@ -163,6 +164,16 @@ def create_content(self, repository, path):
             return requests.post(
                 url, data=data, files=files, **self.request_params)
 
+    def delete_content(self, repository, artifacts):
+        """
+        Delete a list of artifacts from a repository
+        https://pulpproject.org/pulp_rpm/restapi/#tag/Repositories:-Rpm/operation/repositories_rpm_rpm_modify
+        """
+        path = os.path.join(repository, "modify/")
+        url = self.config["base_url"] + path
+        data = {"remove_content_units": artifacts}
+        return requests.post(url, json=data, **self.request_params)
+
     def delete_repository(self, repository):
         """
         Delete an RPM repository
@@ -178,3 +189,30 @@ def delete_distribution(self, distribution):
         """
         url = self.config["base_url"] + distribution
         return requests.delete(url, **self.request_params)
+
+    def wait_for_finished_task(self, task, timeout=86400):
+        """
+        Pulp task (e.g. creating a publication) can be running for an
+        unpredictably long time. We need to wait until it is finished to know
+        what it actually did.
+        """
+        start = time.time()
+        while True:
+            response = self.get_task(task)
+            if not response.ok:
+                break
+            if response.json()["state"] not in ["waiting", "running"]:
+                break
+            if time.time() > start + timeout:
+                break
+            time.sleep(5)
+        return response
+
+    def list_distributions(self, prefix):
+        """
+        Get a list of distributions whose names match a given prefix
+        https://pulpproject.org/pulp_rpm/restapi/#tag/Distributions:-Rpm/operation/distributions_rpm_rpm_list
+        """
+        url = self.url("api/v3/distributions/rpm/rpm/?")
+        url += urlencode({"name__startswith": prefix})
+        return requests.get(url, **self.request_params)