Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
With the new CheckForm testing meta framework which added significantly to our testing coverage and by extension our testing overhead, we were starting to see increasingly flaky tests in the CI/CD pipelines as well as when running them locally (I had recently resorted to getting a fan to cool my laptop when running them as it was heating up significantly...).
Our testing approach is 100% the correct way we should be doing testing so we shouldn't shy away from writing long-running integration tests as the alternatives are either:
Solution
I noticed the flaky tests always seemed to be the ones involving the TCP section of requests (HTTP, gRPC, TCP) and that they took a long time to run and then timed out. As jest tests run in separate worker pools it often seemed like when these TCP tests were running they could cause other tests running at the same time to error out randomly too -- presumably because they were hogging so many CPU resources the other tests were suffering.
Recently in another PR, I noticed that our schema validations are running multiple times per onChange event when a user is typing. It is still something worth investigating at a future date but it makes no discernible impact to real users utilising our application.
However it occurred to me that the TCP fields require typing in valid PEM certificates and keys. The testing key we use is 1750 characters long. If the validation is running 4 or 5 times per character stroke each of these tests is having to validate the input ~10,000 times on top of all the additional computations the input / validation would trigger.
The solution is in the checkform testing helper to override the default
user.type
method and actually turn it into a focus / paste the content so it only triggers the input onChange handlers once. I was considering only making this change to the TCP fields with the large inputs but I couldn't think of a good reason why we shouldn't add the override at this level (there's an argument we should even add it to our custom render function intest/render
?).I've also done a little bit of cleanup and removed the custom jest.timeout declarations and ensured the
waitFor
methods in tests wait much longer than their default 1 second to ensure our test flakiness is kept to a minimum.Before
Had to have a fan running not to overheat my computer 🥵
After
No more fan 😎