Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop repo client tests #1087

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Drop repo client tests #1087

wants to merge 1 commit into from

Conversation

corps
Copy link
Member

@corps corps commented Aug 28, 2024

This tests have several problems.

  1. As designed, they run code paths that depend on the environment. When the environment has certain values, entirely different code paths activate.
  2. This matters because the presence of a GITHUB_TOKEN actually bypasses certain logic altogether.
  3. Github Client as written is very unmodular and hyper overloaded. Even just constructing the object actually invokes lots of external side effects that require mocking, meaning it is hard to truly validate or test independent semantics or behaviors.
  4. As a result, most of these tests involve so many mocks and so much overloaded code, any change in logic or environment tends to result in false positives, while also missing real breakage (such as if the client libraries are upgraded, or the API semantics change, or we use new APIs to change login semantics, or we use a different authentication mechanism)

Action items could include:

  1. Refactoring much of this code to extract separate areas of responsibility that are tested in isolation, including
    a. separating out the logic to decide how auth is selected based on inputs
    b. separating out the mechanism for determining if an environment is configured with valid read or write access
    c. filtering conditions or file system loading logic placed on top of basic github interactions
  2. Using vcr or other integration tooling to use true interactions with the github api as the basis to validate downstream logic, now that they are clearly separated and can be addressed in isolation on the back of a real interaction (and not just a mock).

or

  1. Remove tests
  2. Hope we remember to validate all interactions locally

The first one prioritizes robustness, but requires us to make both an investment now and ongoing investment in thinking about testing frameworks. The second one prioritizes code iteration now, but will break now and in the future, on the hopes that the value of preventing breakage or failures is higher later than it is now.

@corps corps requested a review from a team August 28, 2024 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant