Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Content downloader tests #897

Merged
merged 15 commits into from
Oct 27, 2023
Merged

Content downloader tests #897

merged 15 commits into from
Oct 27, 2023

Commits on Oct 25, 2023

  1. jobs: Fix remotecontentimport node_ids semantics

    As a convenience, the generateed `remotecontentimport` task was setting
    `node_ids` to an empty list if it wasn't specified. However, an empty
    list and `None` are 2 different cases. An empty list is used when no
    node IDs are to be imported, and `None` is used when all node IDs are to
    be imported. Even though we're not actually downloading all nodes here,
    fix the semantics and require that the thumbnails tasks explicitly
    request no nodes. This will be used later in testing.
    dbnicholson committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    780f18c View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. jobs: Return enqueued task from enqueue_next_background_task

    This will be used in testing to check that a task was actually enqueued.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    1d8215c View commit details
    Browse the repository at this point in the history
  2. jobs: Lock database when enqueueing background task

    The storage hook depends on the `BackgroundTask` containing the current
    job ID. However, it's possible the job will change state and the hook
    will run before the job ID has been saved. That would prevent the hook
    from properly synchronizing the updated job state. To prevent that, lock
    the database while the task is being enqueued until the job ID is saved.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    ba915b9 View commit details
    Browse the repository at this point in the history
  3. tests: Improve pytest failure logging

    By default set the log level to INFO so there's useful information on
    failures. Task handling is multithreaded, so add the thread name to help
    debugging.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    3e97462 View commit details
    Browse the repository at this point in the history
  4. ci: Run pytest tests with debug logging

    The logs won't be shown if the tests pass, but if they don't the
    messages can be invaluable for debugging failures.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    4b95c14 View commit details
    Browse the repository at this point in the history
  5. tests: Disable ping tasks

    Learning Equality obviously has no use for the test suite constantly
    telling it about an ephemeral deployment. More importantly, the ping
    task can hang and interfere with our own tasks being scheduled.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    9f20fa5 View commit details
    Browse the repository at this point in the history
  6. tests: Generate fake channel databases

    In order to exercise the collection downloader, we need channel data.
    The `create_channeldb_json.py` script and `db.json.template` file are
    used to create JSON files representing fake channels. The generated JSON
    files contain the data necessary to create sqlite channel databases with
    sqlalchemy as well as the content data inlined. 18 fake channels have
    been created and will be used in later commits.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    6c3180a View commit details
    Browse the repository at this point in the history
  7. tests: Add fixture to create content directory

    The `create_contentdir` function takes the fake channel DB JSON files
    and creates a content directory with the channel databases and content
    files for testing. That's made available as the `contentdir` pytest
    fixture.
    
    A standalone script is provided as a convenience and for exercising the
    functionality outside of the test suite.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    1d4984b View commit details
    Browse the repository at this point in the history
  8. tests: Add content server fixture

    This runs an HTTP server for the test content so that we can import
    channels and content during tests just like they were being imported
    from studio.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    f59744b View commit details
    Browse the repository at this point in the history
  9. tests: Add test collections

    Provide test collections using all the packs in the current
    endless-key-collections release. The fake packs have very regular
    structures and use the fake channel data.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    bf986a6 View commit details
    Browse the repository at this point in the history
  10. tests: Add collectionviews API tests

    These exercise all of the API endpoints that don't interact with the
    download manager.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    c312ec3 View commit details
    Browse the repository at this point in the history
  11. tests: Add fixtures for channel and content imports

    This adds a few pytest fixtures for running channel and content import
    jobs. Beyond creating a facility and a facility user with appropriate
    permissions, 2 fixtures for handling interactions between Django and
    SQLAlchemy. Some of this is lifted from Kolibri and/or depends on
    Kolibri internals. We'll see how well they hold up over time.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    d1e4e56 View commit details
    Browse the repository at this point in the history
  12. tests: Add jobs tests

    While here, add a few log messages to aid when debugging failures.
    dbnicholson committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    f3973d2 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. Add pytest-rerunfailures dev dependency

    It seems that Kolibri's task worker can hang occasionally, so use
    pytest-rerunfailures to mark tests that can be retried on failure.
    dbnicholson committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    a73de4c View commit details
    Browse the repository at this point in the history
  2. tests: Add download manager test

    This is a basic smoketest that the collection downloader can be run with
    any collection specified. Unfortunately, Kolibri's task worker seems to
    hang sometimes during either the foreground or background downloading.
    Use pytest-rerunfailures's `flaky` mark to try again when that happens.
    
    Fixes: #778
    dbnicholson committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    f8563e9 View commit details
    Browse the repository at this point in the history