Re-implement acts as Ansible playbooks

[ingvagabund]

Correspondence:

• act = Ansible playbook ( more flexible representation )
• plugin = Ansible module ( 1:1 correspondence in most cases )

Plugins are currently implemented as python modules so it should be pretty straightforward to wrap them as Ansible modules. The data exchanged between plugins are JSON based. The same can be used in the playbooks.

Then:

• retrievers as Ansible modules (saying what I want, not how I get it)
• artefact readers/writters as Ansible modules (saying does it exists/no how to find it exists, say store it/not how or where)

[ingvagabund]

ScanDistributionBuild act transformed into playbook:

---
product: Fedora
distribution: f27
build: etcd-3.2.7-1.fc27
rpms:
  - etcd-3.2.7-1.fc27.src.rpm
  - etcd-3.2.7-1.fc27.aarch64.rpm
  - etcd-unit-test-3.2.7-1.fc27.aarch64.rpm
  - etcd-debuginfo-3.2.7-1.fc27.aarch64.rpm
  - etcd-debugsource-3.2.7-1.fc27.aarch64.rpm
srpm: etcd-3.2.7-1.fc27.src.rpm

# Parse src rpm first
- name: Check if ARTEFACT_GOLANG_PROJECT_DISTRIBUTION_PACKAGES is already processed
  artefactreader:
    key:
      artefact: ARTEFACT_GOLANG_PROJECT_INFO_FEDORA
      product: "{{ product }}"
      distribution: "{{ distribution }}"
      build: "{{ build }}"
      rpm: "{{ srpm }}"
  register: srpm_check

- when: srpm_check.exists
  srpminfo: "{{ srpm_check.artefact }}"

- when: not srpm_check.exists
  block:
  - name: Retrieve srpm build
    rpmretriever:
      product: "{{ product }}"
      distribution: "{{ distribution }}"
      build: "{{ build }}"
      rpm: "{{ srpm }}"
    register: srpminfo

  - name: Extract data from rpm
    specdataextractor:
      product: "{{ product }}"
      distribution: "{{ distribution }}"
      build: "{{ build }}"
      srpm: "{{ srpm }}"
      directory: srpminfo.directory
    register: srpmartefacts

  - name: Store rpm artefacts
    artefactwriter:
      artefact: "{{ srpmartefacts[ARTEFACT_GOLANG_PROJECT_INFO_FEDORA] }}"

# rpm expected as a json {"name": ..., "arch": ...}
- with_items: "{{ rpms }}"
  when: item.name | select("endswith", "devel.rpm")
  # Loop through all rpms
  block:
  - name: Check if ARTEFACT_GOLANG_PROJECT_DISTRIBUTION_PACKAGES is already processed
    artefactreader:
      key:
        artefact: ARTEFACT_GOLANG_PROJECT_DISTRIBUTION_PACKAGES
        product: "{{ product }}"
        distribution: "{{ distribution }}"
        build: "{{ build }}"
        rpm: "{{ rpm_item }}"
    register: packages_checks

  - name: Check if ARTEFACT_GOLANG_PROJECT_DISTRIBUTION_EXPORTED_API is already processed
    artefactreader:
      key:
        artefact: ARTEFACT_GOLANG_PROJECT_DISTRIBUTION_EXPORTED_API
        product: "{{ product }}"
        distribution: "{{ distribution }}"
        build: "{{ build }}"
        rpm: "{{ rpm_item }}"
    register: api_checks

  - when: not packages_checks.exists || not api_checks.exists
    block:
    - name: Retrieve rpm build
      rpmretriever:
        product: "{{ product }}"
        distribution: "{{ distribution }}"
        build: "{{ build }}"
        rpm: "{{ rpm_item }}"
      register: rpminfo

    - name: Extract data from rpm
      distributiongosymbolsextractor:
        product: "{{ product }}"
        distribution: "{{ distribution }}"
        build: "{{ build }}"
        rpm: "{{ rpm_item }}"
        directory: rpminfo.directory
      register: rpmartefacts

    - name: Create ipprefix2rpm mapping
      ipprefix2rpmmapper:
        golangprojectdistributionpackages: "{{ rpmartefacts[ARTEFACT_GOLANG_PROJECT_DISTRIBUTION_PACKAGES] }}"
        golangprojectinfofedora: "{{ srpmartefacts[ARTEFACT_GOLANG_PROJECT_INFO_FEDORA] }}"
      register: ipp2rpm

    - name: Store rpm artefacts
      artefactwriter:
        artefact: "{{ item }}"
      with_items:
      - rpmartefacts[ARTEFACT_GOLANG_PROJECT_DISTRIBUTION_PACKAGES]
      - rpmartefacts[ARTEFACT_GOLANG_PROJECT_DISTRIBUTION_EXPORTED_API]
      - ipp2rpm.artefact

[ingvagabund]

@jcajka FYI

[ingvagabund]

Following Ansible modules will be required:

• artefactreader: reads an artefact from a storage
• artefactwriter: writes an artefact into a storage
• rpmretriever: retrieves rpms from Koji/Brew
• specdataextractor: plugin responsible for extracting data from a spec file
• distributiongosymbolsextractor: plugin responsible for extracting data from an rpm build
• ipprefix2rpmmapper: plugin combining two artefacts into one

So we just combine I/O of Ansible modules/plugins together. This was the original idea: combine black boxes on a declarative level so noone has to be familiar with the underlying infrastructure. Given the nature of the Ansible, it is the perfect DSL candidate.

Most of the code is already implemented in the existing infrastructure. So it should be more or less copy-pasta and some minimal coding.

[ingvagabund]

@fridex in case you are bored :)

[ingvagabund]

There are more complex acts (e.g. ScanUpstreamRepository) that require more logic to be hidden in Ansible modules/filters. Hope it will not be such a PITA to transform it either.

[ingvagabund]

So far it looks awesome (https://github.com/gofed/infra/tree/migrate-to-ansible/ansible):

---
- name: Go code inspection act
  connection: local
  hosts: localhost
  roles:
  - role: gofedinfra
  tasks:
  - name: Retrieve source code from a repository
    repositorycoderetriever:
      repository: github.com/coreos/etcd/cmd/etcdctl
      hexsha: 121edf0467052d55876a817b89875fb39a99bf78
    register: output

  - debug: var=output

  - name: Extract artefacts from a Go code
    gosymbolsextractor:
      directory: "{{ output.directory }}"
      repository: github.com/coreos/etcd
      hexsha: 121edf0467052d55876a817b89875fb39a99bf78
      ipprefix: github.com/coreos/etcd
    register: eoutput

  - name: Remove working directory
    file:
      name: "{{ output.directory }}"
      state: absent

now just add artefact reader/writter and it's done.

[ingvagabund]

Layout the playbooks in the following manner:

language/category/jobname.yml

E.g.

• go/codeanalysis/apiextractor.yml
• go/codeanalysis/listextractor.yml