docs: adapt documentation after infrastructure code split

CONTRIBUTING.rst

@@ -19,33 +19,26 @@ portal developments, you can `watch ongoing discussions
 `become part of the team
 <https://github.com/orgs/cernopendata/teams>`_.
 
-Code contributions
-------------------
+Contributions
+-------------
 
 We follow typical `GitHub flow
-<https://guides.github.com/introduction/flow/index.html>`_.
+<https://docs.github.com/en/get-started/using-github/github-flow>`_.
 
 1. Fork this repository into your personal space.
-2. Start a new topical branch for any contribution.  Name it sensibly,
-   say ``fix-event-display-icons``.
-3. Test your branch on a local site.  If everything works as expected,
-   please `sign your commits
-   <https://invenio.readthedocs.io/en/latest/technology/git.html#r2-remarks-on-commit-log-messages>`_
-   to indicate its quality.
-4. Create `logically separate commits for logically separate things
-   <https://invenio.readthedocs.io/en/latest/technology/git.html#r1-remarks-on-commit-history>`_.
-   Check out our usual `development practices
-   <https://invenio-github.readthedocs.io/en/latest/contributing.html>`_.
-5. Please add any ``(closes #123)`` directives in your commit log
-   message if your pull request closes an open issue.
-6. Issue a pull request.  If the branch is not quite ready yet, please
-   indicate ``WIP`` (=work in progress) in the pull request title.
-
-For more information on how we work with branches, see our `developing
-guide <DEVELOPING.rst>`_.
-
-Chatroom
---------
-
-Our chatroom is `on gitter
-<https://gitter.im/cernopendata/opendata.cern.ch>`_.
+2. Start a new topical branch for any contribution. Name it descriptively, for
+   example ``fix-cms-2012-collision-energy``.
+3. Ideally, test your branch on a local development site to see if everything
+   works correctly.
+4. Open a pull request against this repository. Verify that all the continuous
+   integration checks are passing. If not, please amend the pull request
+   accordingly.
+
+For more information on the process, please see our `developing guide
+<DEVELOPING.rst>`_.
+
+Support
+-------
+
+You can also get in touch via our `Mattermost
+<https://mattermost.web.cern.ch/it-dep/channels/opendata>`_ chat room.

DEVELOPING.rst

@@ -5,66 +5,130 @@
 .. contents::
    :backlinks: none
 
-Installation
-============
+This document describes how you can run a local instance of the CERN Open Data
+portal in order to work with the content records and associated documentation.
+
+Prerequisites
+=============
+
+You will need to clone two repositories:
 
-This module contains the content of the CERN Open Data instance. It does not contain the code of the portal itself.
-For local development, it is recommended to have an installation of the CERN Open Data Portal. For detail instructions
-on how to install the portal, please follow `these instructions <https://github.com/cernopendata/cernopendata-portal/blob/main/DEVELOPING.rst>`_.
+- `opendata.cern.ch <https://github.com/cernopendata/opendata.cern.ch>`_ which
+  contains the open data content;
+
+- `cernopendata-portal <https://github.com/cernopendata/cernopendata-portal>`_
+  which contains the portal infrastructure code.
+
+Please make sure to also install `Docker
+<https://docs.docker.com/get-started/get-docker/>`_ and `Docker Compose
+<https://docs.docker.com/compose/install/>`_ v2 that is used for local
+developments.
 
 Quick start
--------------------
-For a quickstart guide, do the following:
+===========
+
+In order to create a local CERN Open Data portal instance, please proceed as
+follows:
 
 .. code-block:: console
 
-  $ # Checkout this repository
-  $ git clone https://github.com/cernopendata/opendata.cern.ch.git
-  $ # Checkout the module with the portal
-  $ git clone https://github.com/cernopendata/cernopendata-portal.git
-  $ # Move to the directory of the content
+  $ git clone https://github.com/johndoe/opendata.cern.ch
+  $ git clone https://github.com/johndoe/cernopendata-portal
   $ cd opendata.cern.ch
-  $ # Make sure that the latest images are available
   $ docker compose pull
-  $ # Start the services
   $ docker compose up -d
-  $ # Give enough time to the containers to start properly. Note that there are some dependencies among them,
-  $ # and the web container starts by setting up the development environment
-  $ sleep 120
-  $ # Create the basic structure
-  $ docker exec -i -t opendatacernch-web-1 /code/scripts/populate-instance.sh --skip-records --skip-docs
-  $ docker exec -i -t opendatacernch-web-1 cernopendata fixtures records \
-              --mode insert-or-replace \
-              -f /content/data/records/cms-primary-datasets.json
-..
+  $ sleep 120 # give enough time for the containers to start properly
+  $ docker exec -i -t opendatacernch-web-1 /code/scripts/populate-instance.sh \
+        --skip-records --skip-docs
+
+This will create a running instance of the CERN Open Data portal with a
+relatively empty content. The portal will be accessible locally at
+`http://127.0.0.1:500 <http://127.0.0.1:5000>`_.
 
+If you would like to stop and delete your local instance, you can do:
 
-At this point, all the services should be up and running. If you go to a web browser to http://0.0.0.0:5000/, you should
-see the web portal, with the vocabularies and some documents about the portal itself.
+.. code-block:: console
 
-From this moment on,
+   $ docker compose down -v
 
-Defining new entries
+Working with records
 ====================
 
-This repository has the following data structure:
+If you would like to work with certain data records and test your edits on your
+local instance, you can proceed as follows.
+
+Edit the record file, such as CMS 2012 collision dataset records:
+
+   .. code-block:: console
 
-* data:
-    * records: Put here the entries that should be inserted as records
-    * docs: This folder will be for the documents
-    * images: And this is for static images that might be needed
-* scripts: Directory with shell scripts to help the development
+  $ vim data/records/cms-primary-datasets.json
+
+Upload the local file into your instance:
+
+.. code-block:: console
+
+  $ docker exec -i -t opendatacernch-web-1 cernopendata fixtures records \
+        --mode insert-or-replace \
+        -f /content/data/records/cms-primary-datasets.json
+
+You can then check your changes at `http://127.0.0.1:500
+<http://127.0.0.1:5000>`_.
+
+Note that you can take advantage of shell scripting if you would like to upload
+all experiment records locally, for example for ATLAS:
+
+.. code-block:: console
+
+  $ for file in data/records/atlas-*; do \
+       docker exec -i -t opendatacernch-web-1 cernopendata fixtures records \
+           --mode insert-or-replace -f $file; \
+    done
+
+Understanding metadata fields
+=============================
+
+When working with data records, there are several fields such as
+`collision_energy` that you can use to store the content. The list of all
+available record fields, together with their semantic meaning, is described in
+the JSON Schema files. You can find the `record schema
+<https://github.com/cernopendata/cernopendata-portal/blob/main/cernopendata/jsonschemas/records/record-v1.0.0.json>`_
+in the portal infrastructure repository.
+
+If you would like to modify the JSON schema, for example to add a new field,
+this would require working with the `cernopendata-portal` sister repository.
+Please see its own `documenation
+<https://github.com/cernopendata/cernopendata-portal/>`_ about how to add new
+metadata fields. We would be happy to assist with the process.
+
+Understanding output templates
+==============================
+
+If you would like to change the way how the data records are displayed on the
+web, for example to introduce new section displaying newly added field, this is
+something that is governed by `Jinja templating language
+<https://jinja.palletsprojects.com/en/2.10.x/templates/>`_ in the
+`cernopendata-portal` sister repository. Please see its own `documenation
+<https://github.com/cernopendata/cernopendata-portal/>`_ about how to amend
+look and feel of the record metadata. We would be happy to assist with the
+process.
+
+Verifying metadata conformance
+==============================
+
+You can use the provided helper script `check_fixtures.py` to check the
+conformance of record files to the required minimal standard:
+
+.. code-block:: console
 
-If you want to modify the json schema, mappings or templates, you will find these folders in the
-`cernopendata portal <https://github.com/cernopendata/cernopendata-portal/>`_ repository
+   ./scripts/check_fixtures.py
 
 Working with docs/records
 -------------------------
 
 The recommended development process is the following:
 
-1. Create the entries under data/(records/docs)
-2. Validate that the yaml syntax is correct
+1. Create the entries under data/(records/docs) 2. Validate that the yaml
+   syntax is correct
 
 .. code-block:: console
 

README.rst

@@ -8,34 +8,36 @@
 .. image:: https://img.shields.io/badge/licence-GPL_2-green.svg?style=flat
    :target: https://raw.githubusercontent.com/cernopendata/opendata.cern.ch/master/LICENSE
 
-.. image:: https://badges.gitter.im/Join%20Chat.svg
-   :target: https://gitter.im/cernopendata/opendata.cern.ch?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge
-
 About
 -----
 
-This is the source code behind CERN Open Data portal. You can access the portal
-at `http://opendata.cern.ch/ <http://opendata.cern.ch>`_.  The source code uses
-`Invenio <https://inveniosoftware.org/>`_ digital repository framework.
+This repository is part of for the `CERN Open Data portal
+<https://opendata.cern.ch>`_, hosting the source versions of the content,
+notably data records and associated documentation pages. The CERN Open Data
+portal's infrastructure code is living in the sister `cernopendata-portal
+<https://github.com/cernopendata/cernopendata-portal>`_ repository. The CERN
+Open Data portal is built upon the `Invenio <https://inveniosoftware.org/>`_
+digital repository framework.
 
 Developing
 ----------
 
-If you'd like to install a demo site locally for personal developments, please
-see `developing guide <DEVELOPING.rst>`_ for more information.
+If you'd like to install a demo site of the CERN Open Data portal locally for
+personal developments, please see the `developing guide <DEVELOPING.rst>`_ for
+more information.
 
 Contributing
 ------------
 
-Bug reports, feature requests and code contributions are encouraged and
-welcome!  Please see `contributing guide <CONTRIBUTING.rst>`_ for more
-information.
+Bug reports, feature requests and contributions are encouraged and welcome!
+Please see the `contributing guide <CONTRIBUTING.rst>`_ for more information.
 
 Support
 -------
 
 You can ask questions at our `Forum <https://opendata-forum.cern.ch/>`_ or get
-in touch via our `Chatroom <https://gitter.im/cernopendata/opendata.cern.ch>`_.
+in touch via our `Mattermost
+<https://mattermost.web.cern.ch/it-dep/channels/opendata>`_ chat room.
 
 Authors
 -------

scripts/check_fixtures.py

@@ -25,7 +25,7 @@ def print_warning(filename, recid="", field="", message="missing-field"):
 def main():
     """Check record fixtures for basic fields."""
     problems_found = False
-    fixtures_directory = "cernopendata/modules/fixtures/data/records"
+    fixtures_directory = "data/records"
     for filename in os.listdir(fixtures_directory):
         records = json.loads(open(fixtures_directory + os.sep + filename, "r").read())
         for record in records: