Derive the `session.id`/`order_number` from the `upload.id` #720

Swatinem · 2024-09-18T12:14:53Z

This re-applies #713, which was reverted previously, as production upload.id exceed the bit width of upload.order_number.

The problem was that order_number is a 32-bit signed int (effectively 31-bit),
and duplicating the reports_upload.id into it does not work, as that value already exceeds 31-bits in production.

This tries to work around that problem by deriving the order_number/session.id like so:

Take the lower 16-bit of the upload.id
Shift those to occupy bits 12-28 in the derived order_number

This way, the derived ID should not collide with a carry-forwarded session with order_numbers up to 12 bits.
And the final derived/deterministic order_number is also below the 32 (effectively 31) bit threshold.

codecov-staging · 2024-09-18T12:21:07Z

Codecov Report

Attention: Patch coverage is 67.16418% with 22 lines in your changes missing coverage. Please review.

Files	Patch %	Lines
tasks/tests/integration/test_upload_e2e.py	8.33%	11 Missing ⚠️
tasks/upload_finisher.py	20.00%	8 Missing ⚠️
services/report/__init__.py	85.71%	1 Missing ⚠️
services/report/raw_upload_processor.py	94.44%	1 Missing ⚠️
tasks/upload_processor.py	83.33%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (6ecd3ba) and HEAD (cdf4bba). Click for more details.

HEAD has 4 uploads less than BASE

Flag BASE (6ecd3ba) HEAD (cdf4bba)

latest-uploader-overall 2 1

unit 2 1

integration 2 0

@@            Coverage Diff             @@
##             main     #720      +/-   ##
==========================================
- Coverage   98.07%   90.63%   -7.44%     
==========================================
  Files         434      434              
  Lines       36734    36673      -61     
==========================================
- Hits        36027    33240    -2787     
- Misses        707     3433    +2726

Flag	Coverage Δ
integration	`?`
latest-uploader-overall	`90.63% <67.16%> (-7.44%)`	⬇️
unit	`90.63% <67.16%> (-7.44%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
NonTestCode	`90.23% <78.43%> (-5.75%)`	⬇️
OutsideTasks	`95.46% <94.44%> (-2.63%)`	⬇️

Files	Coverage Δ
database/models/reports.py	`99.01% <100.00%> (-0.50%)`	⬇️
helpers/parallel_upload_processing.py	`23.80% <ø> (-61.91%)`	⬇️
helpers/reports.py	`92.30% <100.00%> (-7.70%)`	⬇️
services/report/tests/unit/test_process.py	`99.07% <100.00%> (ø)`
tasks/tests/unit/test_upload_processing_task.py	`73.70% <100.00%> (-26.30%)`	⬇️
tasks/tests/unit/test_upload_task.py	`54.07% <100.00%> (-45.02%)`	⬇️
tasks/upload.py	`70.98% <100.00%> (-25.35%)`	⬇️
services/report/__init__.py	`89.84% <85.71%> (-6.94%)`	⬇️
services/report/raw_upload_processor.py	`95.76% <94.44%> (+1.82%)`	⬆️
tasks/upload_processor.py	`90.41% <83.33%> (-9.01%)`	⬇️
... and 2 more

... and 54 files with indirect coverage changes

codecov-qa · 2024-09-18T12:21:19Z

❌ 8 Tests Failed:

Tests completed	Failed	Passed	Skipped
1684	8	1676	0

View the top 3 failed tests by shortest run time

tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask test_upload_task_process_individual_report_with_notfound_report_no_retries_yet

Stack Traces | 0.044s run time

self = &lt;worker.tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask object at 0x7f2dcef0adb0&gt;
mocker = &lt;pytest_mock.plugin.MockFixture object at 0x7f2da0e5de20&gt;

    def test_upload_task_process_individual_report_with_notfound_report_no_retries_yet(
        self, mocker
    ):
        # throw an error thats retryable:
        mocker.patch.object(
            ReportService,
            "parse_raw_report_from_storage",
            side_effect=FileNotInStorageError(),
        )
        task = UploadProcessorTask()
        with pytest.raises(Retry):
&gt;           task.process_individual_report(
                ReportService({}),
                CommitFactory.create(),
                Report(),
                UploadFactory.create(),
                RawReportInfo(),
            )

.../tests/unit/test_upload_processing_task.py:757: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
tasks/upload_processor.py:381: in process_individual_report
    processing_result = report_service.build_report_from_raw_content(
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:888: in build_report_from_raw_content
    id=session_id_from_upload_id(upload.id),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

upload_id = None

    def session_id_from_upload_id(upload_id: int) -&gt; int:
        """
        Creates a unique `session_id` from an `upload_id`.
    
        The purpose of the `session_id` is to unique within one report.
        Turning the `upload_id` into the `session_id` trivially achieves this, as
        the `upload_id` is a unique database id.
    
        However in production, the `upload_id` exceeds 31 bits, which overflows
        the datatype of the `session_id`, which is a signed 32-bit integer
        (effectively 31-bits).
    
        We work around this problem by just masking off the high bits, essentially
        wrapping around at that number, and shifting a bit to avoid collisions
        with existing low session numbers.
        """
&gt;       return (upload_id &amp; (2**16 - 1)) &lt;&lt; 12
E       TypeError: unsupported operand type(s) for &amp;: 'NoneType' and 'int'

helpers/reports.py:39: TypeError

services.tests.test_report.TestReportService test_save_full_report

Stack Traces | 0.064s run time

self = &lt;worker.services.tests.test_report.TestReportService object at 0x7f2dd0f01640&gt;
dbsession = &lt;sqlalchemy.orm.session.Session object at 0x7f2dc48899d0&gt;
mock_storage = &lt;shared.storage.memory.MemoryStorageService object at 0x7f2dc46d02c0&gt;
sample_report = &lt;Report files=2&gt;
mock_configuration = &lt;shared.config.ConfigHelper object at 0x7f2dc487d400&gt;

    def test_save_full_report(
        self, dbsession, mock_storage, sample_report, mock_configuration
    ):
        mock_configuration.set_params(
            {
                "setup": {
                    "save_report_data_in_storage": {
                        "only_codecov": False,
                        "report_details_files_array": True,
                    },
                }
            }
        )
        commit = CommitFactory.create()
        dbsession.add(commit)
        dbsession.flush()
        current_report_row = CommitReport(commit_id=commit.id_)
        dbsession.add(current_report_row)
        dbsession.flush()
        report_details = ReportDetails(report_id=current_report_row.id_)
        dbsession.add(report_details)
        dbsession.flush()
        sample_report.sessions[0].archive = ".../to/upload/location"
        sample_report.sessions[
            0
        ].name = "this name contains more than 100 chars 1111111111111111111111111111111111111111111111111111111111111this is more than 100"
        report_service = ReportService({})
&gt;       res = report_service.save_full_report(commit, sample_report)

services/tests/test_report.py:3547: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12.../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:1186: in save_full_report
    self._attach_flags_to_upload(upload, session.flags if session.flags else [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = &lt;services.report.ReportService object at 0x7f2dc7f4d940&gt;
upload = &lt;database.models.reports.Upload object at 0x7f2dc44e87d0&gt;
flag_names = ['unit']

    def _attach_flags_to_upload(self, upload: Upload, flag_names: Sequence[str]):
        """Internal function that manages creating the proper `RepositoryFlag`s and attach the sessions to them
    
        Args:
            upload (Upload): Description
            flag_names (Sequence[str]): Description
    
        Returns:
            TYPE: Description
        """
        all_flags = []
        db_session = upload.get_db_session()
&gt;       repoid = upload.report.commit.repoid
E       AttributeError: 'NoneType' object has no attribute 'commit'

services/report/__init__.py:331: AttributeError

services.tests.test_report.TestReportService test_initialize_and_save_report_needs_backporting

Stack Traces | 0.069s run time

self = &lt;worker.services.tests.test_report.TestReportService object at 0x7f2dd10f5730&gt;
dbsession = &lt;sqlalchemy.orm.session.Session object at 0x7f2dc488d580&gt;
sample_commit_with_report_big = Commit&lt;659c950c1d6ea9da26848c0b2a9010906f68034d@repo&lt;582&gt;&gt;
mock_storage = &lt;shared.storage.memory.MemoryStorageService object at 0x7f2dc485ad80&gt;
mocker = &lt;pytest_mock.plugin.MockFixture object at 0x7f2dc48d17c0&gt;

    def test_initialize_and_save_report_needs_backporting(
        self, dbsession, sample_commit_with_report_big, mock_storage, mocker
    ):
        commit = sample_commit_with_report_big
        report_service = ReportService({})
        mocker.patch.object(
            ReportDetails, "_should_write_to_storage", return_value=True
        )
&gt;       r = report_service.initialize_and_save_report(commit)

services/tests/test_report.py:4240: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:268: in initialize_and_save_report
    self.save_full_report(commit, actual_report)
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:1186: in save_full_report
    self._attach_flags_to_upload(upload, session.flags if session.flags else [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = &lt;services.report.ReportService object at 0x7f2dc48d3860&gt;
upload = &lt;database.models.reports.Upload object at 0x7f2dc46d2600&gt;
flag_names = []

    def _attach_flags_to_upload(self, upload: Upload, flag_names: Sequence[str]):
        """Internal function that manages creating the proper `RepositoryFlag`s and attach the sessions to them
    
        Args:
            upload (Upload): Description
            flag_names (Sequence[str]): Description
    
        Returns:
            TYPE: Description
        """
        all_flags = []
        db_session = upload.get_db_session()
&gt;       repoid = upload.report.commit.repoid
E       AttributeError: 'NoneType' object has no attribute 'commit'

services/report/__init__.py:331: AttributeError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

codecov-public-qa · 2024-09-18T12:21:32Z

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 1684 tests with 8 failed, 1676 passed and 0 skipped.

View the full list of failed tests

pytest

Class name: services.tests.test_report.TestReportService
Test name: test_initialize_and_save_report_carryforward_needed

self = <worker.services.tests.test_report.TestReportService object at 0x7f2dd0f006b0>
dbsession = <sqlalchemy.orm.session.Session object at 0x7f2dc47eeab0>
sample_commit_with_report_big = Commit<46c0973f1b2f53ed03cc6030215d95dec726d3e4@repo<580>>
mocker = <pytest_mock.plugin.MockFixture object at 0x7f2dc487fb30>
mock_storage = <shared.storage.memory.MemoryStorageService object at 0x7f2dc47eec00>

    @pytest.mark.django_db(databases={"default"})
    def test_initialize_and_save_report_carryforward_needed(
        self, dbsession, sample_commit_with_report_big, mocker, mock_storage
    ):
        parent_commit = sample_commit_with_report_big
        commit = CommitFactory.create(
            _report_json=None,
            parent_commit_id=parent_commit.commitid,
            repository=parent_commit.repository,
        )
        dbsession.add(commit)
        dbsession.flush()
        yaml_dict = {"flags": {"enterprise": {"carryforward": True}}}
        report_service = ReportService(UserYaml(yaml_dict))
>       r = report_service.initialize_and_save_report(commit)

services/tests/test_report.py:4085: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:281: in initialize_and_save_report
    self.save_full_report(commit, report, report_code)
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:1186: in save_full_report
    self._attach_flags_to_upload(upload, session.flags if session.flags else [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <services.report.ReportService object at 0x7f2dc5be47d0>
upload = <database.models.reports.Upload object at 0x7f2dc48891c0>
flag_names = ['enterprise']

    def _attach_flags_to_upload(self, upload: Upload, flag_names: Sequence[str]):
        """Internal function that manages creating the proper `RepositoryFlag`s and attach the sessions to them
    
        Args:
            upload (Upload): Description
            flag_names (Sequence[str]): Description
    
        Returns:
            TYPE: Description
        """
        all_flags = []
        db_session = upload.get_db_session()
>       repoid = upload.report.commit.repoid
E       AttributeError: 'NoneType' object has no attribute 'commit'

services/report/__init__.py:331: AttributeError

Class name: services.tests.test_report.TestReportService
Test name: test_initialize_and_save_report_needs_backporting

self = <worker.services.tests.test_report.TestReportService object at 0x7f2dd10f5730>
dbsession = <sqlalchemy.orm.session.Session object at 0x7f2dc488d580>
sample_commit_with_report_big = Commit<659c950c1d6ea9da26848c0b2a9010906f68034d@repo<582>>
mock_storage = <shared.storage.memory.MemoryStorageService object at 0x7f2dc485ad80>
mocker = <pytest_mock.plugin.MockFixture object at 0x7f2dc48d17c0>

    def test_initialize_and_save_report_needs_backporting(
        self, dbsession, sample_commit_with_report_big, mock_storage, mocker
    ):
        commit = sample_commit_with_report_big
        report_service = ReportService({})
        mocker.patch.object(
            ReportDetails, "_should_write_to_storage", return_value=True
        )
>       r = report_service.initialize_and_save_report(commit)

services/tests/test_report.py:4240: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:268: in initialize_and_save_report
    self.save_full_report(commit, actual_report)
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:1186: in save_full_report
    self._attach_flags_to_upload(upload, session.flags if session.flags else [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <services.report.ReportService object at 0x7f2dc48d3860>
upload = <database.models.reports.Upload object at 0x7f2dc46d2600>
flag_names = []

    def _attach_flags_to_upload(self, upload: Upload, flag_names: Sequence[str]):
        """Internal function that manages creating the proper `RepositoryFlag`s and attach the sessions to them
    
        Args:
            upload (Upload): Description
            flag_names (Sequence[str]): Description
    
        Returns:
            TYPE: Description
        """
        all_flags = []
        db_session = upload.get_db_session()
>       repoid = upload.report.commit.repoid
E       AttributeError: 'NoneType' object has no attribute 'commit'

services/report/__init__.py:331: AttributeError

Class name: services.tests.test_report.TestReportService
Test name: test_initialize_and_save_report_report_but_no_details_carryforward_needed

self = <worker.services.tests.test_report.TestReportService object at 0x7f2dd10f49b0>
dbsession = <sqlalchemy.orm.session.Session object at 0x7f2dc48d1ca0>
sample_commit_with_report_big = Commit<af9a73b79e66268620d02676ed1b3073c65bfc7b@repo<581>>
mock_storage = <shared.storage.memory.MemoryStorageService object at 0x7f2dc488dfd0>

    @pytest.mark.django_db(databases={"default"})
    def test_initialize_and_save_report_report_but_no_details_carryforward_needed(
        self, dbsession, sample_commit_with_report_big, mock_storage
    ):
        parent_commit = sample_commit_with_report_big
        commit = CommitFactory.create(
            _report_json=None,
            parent_commit_id=parent_commit.commitid,
            repository=parent_commit.repository,
        )
        dbsession.add(commit)
        dbsession.flush()
        report_row = CommitReport(commit_id=commit.id_)
        dbsession.add(report_row)
        dbsession.flush()
        yaml_dict = {"flags": {"enterprise": {"carryforward": True}}}
        report_service = ReportService(UserYaml(yaml_dict))
>       r = report_service.initialize_and_save_report(commit)

services/tests/test_report.py:4167: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:281: in initialize_and_save_report
    self.save_full_report(commit, report, report_code)
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:1186: in save_full_report
    self._attach_flags_to_upload(upload, session.flags if session.flags else [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <services.report.ReportService object at 0x7f2dc488c470>
upload = <database.models.reports.Upload object at 0x7f2dbbf12210>
flag_names = ['enterprise']

    def _attach_flags_to_upload(self, upload: Upload, flag_names: Sequence[str]):
        """Internal function that manages creating the proper `RepositoryFlag`s and attach the sessions to them
    
        Args:
            upload (Upload): Description
            flag_names (Sequence[str]): Description
    
        Returns:
            TYPE: Description
        """
        all_flags = []
        db_session = upload.get_db_session()
>       repoid = upload.report.commit.repoid
E       AttributeError: 'NoneType' object has no attribute 'commit'

services/report/__init__.py:331: AttributeError

Class name: services.tests.test_report.TestReportService
Test name: test_save_full_report

self = <worker.services.tests.test_report.TestReportService object at 0x7f2dd0f01640>
dbsession = <sqlalchemy.orm.session.Session object at 0x7f2dc48899d0>
mock_storage = <shared.storage.memory.MemoryStorageService object at 0x7f2dc46d02c0>
sample_report = <Report files=2>
mock_configuration = <shared.config.ConfigHelper object at 0x7f2dc487d400>

    def test_save_full_report(
        self, dbsession, mock_storage, sample_report, mock_configuration
    ):
        mock_configuration.set_params(
            {
                "setup": {
                    "save_report_data_in_storage": {
                        "only_codecov": False,
                        "report_details_files_array": True,
                    },
                }
            }
        )
        commit = CommitFactory.create()
        dbsession.add(commit)
        dbsession.flush()
        current_report_row = CommitReport(commit_id=commit.id_)
        dbsession.add(current_report_row)
        dbsession.flush()
        report_details = ReportDetails(report_id=current_report_row.id_)
        dbsession.add(report_details)
        dbsession.flush()
        sample_report.sessions[0].archive = ".../to/upload/location"
        sample_report.sessions[
            0
        ].name = "this name contains more than 100 chars 1111111111111111111111111111111111111111111111111111111111111this is more than 100"
        report_service = ReportService({})
>       res = report_service.save_full_report(commit, sample_report)

services/tests/test_report.py:3547: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12.../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:1186: in save_full_report
    self._attach_flags_to_upload(upload, session.flags if session.flags else [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <services.report.ReportService object at 0x7f2dc7f4d940>
upload = <database.models.reports.Upload object at 0x7f2dc44e87d0>
flag_names = ['unit']

    def _attach_flags_to_upload(self, upload: Upload, flag_names: Sequence[str]):
        """Internal function that manages creating the proper `RepositoryFlag`s and attach the sessions to them
    
        Args:
            upload (Upload): Description
            flag_names (Sequence[str]): Description
    
        Returns:
            TYPE: Description
        """
        all_flags = []
        db_session = upload.get_db_session()
>       repoid = upload.report.commit.repoid
E       AttributeError: 'NoneType' object has no attribute 'commit'

services/report/__init__.py:331: AttributeError

Class name: tasks.tests.unit.test_preprocess_upload.TestPreProcessUpload
Test name: test_preprocess_task

self = <worker.tasks.tests.unit.test_preprocess_upload.TestPreProcessUpload object at 0x7f2dcf0ad5e0>
mocker = <pytest_mock.plugin.MockFixture object at 0x7f2da1b97f20>
mock_configuration = <shared.config.ConfigHelper object at 0x7f2da1b828d0>
dbsession = <sqlalchemy.orm.session.Session object at 0x7f2da1b81e50>
mock_storage = <shared.storage.memory.MemoryStorageService object at 0x7f2da1b80620>
mock_redis = <MagicMock name='_get_redis_instance_from_url()' id='139833963461856'>
celery_app = <Celery celery.tests at 0x7f2da1a3dc70>
sample_report = <Report files=2>

    @pytest.mark.django_db(databases={"default"})
    def test_preprocess_task(
        self,
        mocker,
        mock_configuration,
        dbsession,
        mock_storage,
        mock_redis,
        celery_app,
        sample_report,
    ):
        # get_existing_report_for_commit gets called for the parent commit
        mocker.patch.object(
            ReportService,
            "get_existing_report_for_commit",
            return_value=sample_report,
        )
        commit_yaml = {
            "flag_management": {
                "individual_flags": [
                    {
                        "name": "unit",
                        "carryforward": True,
                    }
                ]
            }
        }
        mocker.patch(
            "services.repository.fetch_commit_yaml_from_provider",
            return_value=commit_yaml,
        )
        mock_save_commit = mocker.patch(
            "services.repository.save_repo_yaml_to_database_if_needed"
        )
    
        def fake_possibly_shift(report, base, head):
            return report
    
        mock_possibly_shift = mocker.patch.object(
            ReportService,
            "_possibly_shift_carryforward_report",
            side_effect=fake_possibly_shift,
        )
        commit, report = self.create_commit_and_report(dbsession)
    
>       result = PreProcessUpload().process_impl_within_lock(
            dbsession,
            repoid=commit.repository.repoid,
            commitid=commit.commitid,
            report_code=None,
        )

.../tests/unit/test_preprocess_upload.py:62: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tasks/preprocess_upload.py:117: in process_impl_within_lock
    commit_report = report_service.initialize_and_save_report(commit, report_code)
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:281: in initialize_and_save_report
    self.save_full_report(commit, report, report_code)
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:1186: in save_full_report
    self._attach_flags_to_upload(upload, session.flags if session.flags else [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <services.report.ReportService object at 0x7f2da1b944a0>
upload = <database.models.reports.Upload object at 0x7f2da1c26840>
flag_names = ['unit']

    def _attach_flags_to_upload(self, upload: Upload, flag_names: Sequence[str]):
        """Internal function that manages creating the proper `RepositoryFlag`s and attach the sessions to them
    
        Args:
            upload (Upload): Description
            flag_names (Sequence[str]): Description
    
        Returns:
            TYPE: Description
        """
        all_flags = []
        db_session = upload.get_db_session()
>       repoid = upload.report.commit.repoid
E       AttributeError: 'NoneType' object has no attribute 'commit'

services/report/__init__.py:331: AttributeError

Class name: tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask
Test name: test_upload_processor_call_with_upload_obj

self = <worker.tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask object at 0x7f2dcef36810>
mocker = <pytest_mock.plugin.MockFixture object at 0x7f2da14d5970>
mock_configuration = <shared.config.ConfigHelper object at 0x7f2da14a0860>
dbsession = <sqlalchemy.orm.session.Session object at 0x7f2da13d35f0>
mock_storage = <shared.storage.memory.MemoryStorageService object at 0x7f2da13d2660>

    @pytest.mark.django_db(databases={"default"})
    def test_upload_processor_call_with_upload_obj(
        self, mocker, mock_configuration, dbsession, mock_storage
    ):
        mocker.patch.object(
            USE_LABEL_INDEX_IN_REPORT_PROCESSING_BY_REPO_ID,
            "check_value",
            return_value=False,
        )
    
        mocked_1 = mocker.patch.object(ArchiveService, "read_chunks")
        mocked_1.return_value = None
        commit = CommitFactory.create(
            message="dsidsahdsahdsa",
            commitid="abf6d4df662c47e32460020ab14abf9303581429",
            author__service="github",
            repository__owner__unencrypted_oauth_token="testulk3d54rlhxkjyzomq2wh8b7np47xabcrkx8",
            repository__owner__service="github",
            repository__owner__username="ThiagoCodecov",
            repository__name="example-python",
        )
        dbsession.add(commit)
        dbsession.flush()
        current_report_row = CommitReport(commit_id=commit.id_)
        dbsession.add(current_report_row)
        dbsession.flush()
        report_details = ReportDetails(
            report_id=current_report_row.id_, _files_array=[]
        )
        dbsession.add(report_details)
        dbsession.flush()
        url = ".../C3C4715CA57C910D11D5EB899FC86A7E/4c4e4654ac25037ae869caeb3619d485970b6304/a84d445c-9c1e-434f-8275-f18f1f320f81.txt"
        upload = UploadFactory.create(
            report=current_report_row, state="started", storage_path=url
        )
        dbsession.add(upload)
        dbsession.flush()
        with open(here.parent.parent / "samples" / "sample_uploaded_report_1.txt") as f:
            content = f.read()
            mock_storage.write_file("archive", url, content)
        redis_queue = [{"url": url, "upload_pk": upload.id_}]
        mocked_3 = mocker.patch.object(UploadProcessorTask, "app")
        mocked_3.send_task.return_value = True
        result = UploadProcessorTask().process_impl_within_lock(
            db_session=dbsession,
            previous_results={},
            repoid=commit.repoid,
            commitid=commit.commitid,
            commit_yaml={"codecov": {"max_report_age": False}},
            arguments_list=redis_queue,
            report_code=None,
        )
        expected_result = {
            "processings_so_far": [
                {
                    "arguments": {"url": url, "upload_pk": upload.id_},
                    "successful": True,
                }
            ]
        }
        assert expected_result == result
        assert commit.message == "dsidsahdsahdsa"
        assert upload.state == "processed"
        expected_generated_report = {
            "files": {
                "awesome/__init__.py": [
                    0,
                    [0, 14, 10, 4, 0, "71.42857", 0, 0, 0, 0, 0, 0, 0],
                    None,
                    None,
                ],
                "tests/__init__.py": [
                    1,
                    [0, 3, 2, 1, 0, "66.66667", 0, 0, 0, 0, 0, 0, 0],
                    None,
                    None,
                ],
                "tests/test_sample.py": [
                    2,
                    [0, 7, 7, 0, 0, "100", 0, 0, 0, 0, 0, 0, 0],
                    None,
                    None,
                ],
            },
            "sessions": {
                str(upload.id_): {
                    "N": None,
                    "a": url,
                    "c": None,
                    "e": None,
                    "f": [],
                    "j": None,
                    "n": None,
                    "p": None,
                    "t": [3, 24, 19, 5, 0, "79.16667", 0, 0, 0, 0, 0, 0, 0],
                    "u": None,
>                   "d": commit.report_json["sessions"][str(upload.id_)]["d"],
                    "st": "uploaded",
                    "se": {},
                }
            },
        }
E       KeyError: '3185'

.../tests/unit/test_upload_processing_task.py:387: KeyError

Class name: tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask
Test name: test_upload_task_process_individual_report_with_notfound_report

self = <worker.tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask object at 0x7f2dcef0a8d0>
mocker = <pytest_mock.plugin.MockFixture object at 0x7f2da149fce0>
mock_configuration = <shared.config.ConfigHelper object at 0x7f2da17ac410>
dbsession = <sqlalchemy.orm.session.Session object at 0x7f2da17afe90>
mock_repo_provider = <MagicMock name='_get_repo_provider_service_instance()' spec='Github' id='139833956363072'>
mock_storage = <shared.storage.memory.MemoryStorageService object at 0x7f2da14a3f50>
mock_redis = <MagicMock name='_get_redis_instance_from_url()' id='139833959506448'>

    def test_upload_task_process_individual_report_with_notfound_report(
        self,
        mocker,
        mock_configuration,
        dbsession,
        mock_repo_provider,
        mock_storage,
        mock_redis,
    ):
        mocked_1 = mocker.patch.object(ArchiveService, "read_chunks")
        mocked_1.return_value = None
        false_report = mocker.MagicMock(
            to_database=mocker.MagicMock(return_value=({}, "{}")), totals=ReportTotals()
        )
        # Mocking retry to also raise the exception so we can see how it is called
        mocked_4 = mocker.patch.object(UploadProcessorTask, "app")
        mocked_4.send_task.return_value = True
        commit = CommitFactory.create(
            message="",
            commitid="abf6d4df662c47e32460020ab14abf9303581429",
            repository__owner__unencrypted_oauth_token="testulk3d54rlhxkjyzomq2wh8b7np47xabcrkx8",
            repository__owner__username="ThiagoCodecov",
            repository__yaml={"codecov": {"max_report_age": False}},
        )
        dbsession.add(commit)
        dbsession.flush()
        upload = UploadFactory.create(
            report__commit=commit, storage_path="locationlocation"
        )
        dbsession.add(upload)
        task = UploadProcessorTask()
        task.request.retries = 1
>       result = task.process_individual_report(
            report_service=ReportService({"codecov": {"max_report_age": False}}),
            commit=commit,
            report=false_report,
            raw_report_info=RawReportInfo(),
            upload=upload,
        )

.../tests/unit/test_upload_processing_task.py:732: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
tasks/upload_processor.py:381: in process_individual_report
    processing_result = report_service.build_report_from_raw_content(
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:888: in build_report_from_raw_content
    id=session_id_from_upload_id(upload.id),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

upload_id = None

    def session_id_from_upload_id(upload_id: int) -> int:
        """
        Creates a unique `session_id` from an `upload_id`.
    
        The purpose of the `session_id` is to unique within one report.
        Turning the `upload_id` into the `session_id` trivially achieves this, as
        the `upload_id` is a unique database id.
    
        However in production, the `upload_id` exceeds 31 bits, which overflows
        the datatype of the `session_id`, which is a signed 32-bit integer
        (effectively 31-bits).
    
        We work around this problem by just masking off the high bits, essentially
        wrapping around at that number, and shifting a bit to avoid collisions
        with existing low session numbers.
        """
>       return (upload_id & (2**16 - 1)) << 12
E       TypeError: unsupported operand type(s) for &: 'NoneType' and 'int'

helpers/reports.py:39: TypeError

Class name: tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask
Test name: test_upload_task_process_individual_report_with_notfound_report_no_retries_yet

self = <worker.tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask object at 0x7f2dcef0adb0>
mocker = <pytest_mock.plugin.MockFixture object at 0x7f2da0e5de20>

    def test_upload_task_process_individual_report_with_notfound_report_no_retries_yet(
        self, mocker
    ):
        # throw an error thats retryable:
        mocker.patch.object(
            ReportService,
            "parse_raw_report_from_storage",
            side_effect=FileNotInStorageError(),
        )
        task = UploadProcessorTask()
        with pytest.raises(Retry):
>           task.process_individual_report(
                ReportService({}),
                CommitFactory.create(),
                Report(),
                UploadFactory.create(),
                RawReportInfo(),
            )

.../tests/unit/test_upload_processing_task.py:757: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
tasks/upload_processor.py:381: in process_individual_report
    processing_result = report_service.build_report_from_raw_content(
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:888: in build_report_from_raw_content
    id=session_id_from_upload_id(upload.id),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

upload_id = None

    def session_id_from_upload_id(upload_id: int) -> int:
        """
        Creates a unique `session_id` from an `upload_id`.
    
        The purpose of the `session_id` is to unique within one report.
        Turning the `upload_id` into the `session_id` trivially achieves this, as
        the `upload_id` is a unique database id.
    
        However in production, the `upload_id` exceeds 31 bits, which overflows
        the datatype of the `session_id`, which is a signed 32-bit integer
        (effectively 31-bits).
    
        We work around this problem by just masking off the high bits, essentially
        wrapping around at that number, and shifting a bit to avoid collisions
        with existing low session numbers.
        """
>       return (upload_id & (2**16 - 1)) << 12
E       TypeError: unsupported operand type(s) for &: 'NoneType' and 'int'

helpers/reports.py:39: TypeError

codecov · 2024-09-18T12:21:34Z

❌ 8 Tests Failed:

Tests completed	Failed	Passed	Skipped
1684	8	1676	0

View the top 3 failed tests by shortest run time

tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask test_upload_task_process_individual_report_with_notfound_report_no_retries_yet

Stack Traces | 0.044s run time

self = &lt;worker.tasks.tests.unit.test_upload_processing_task.TestUploadProcessorTask object at 0x7f2dcef0adb0&gt;
mocker = &lt;pytest_mock.plugin.MockFixture object at 0x7f2da0e5de20&gt;

    def test_upload_task_process_individual_report_with_notfound_report_no_retries_yet(
        self, mocker
    ):
        # throw an error thats retryable:
        mocker.patch.object(
            ReportService,
            "parse_raw_report_from_storage",
            side_effect=FileNotInStorageError(),
        )
        task = UploadProcessorTask()
        with pytest.raises(Retry):
&gt;           task.process_individual_report(
                ReportService({}),
                CommitFactory.create(),
                Report(),
                UploadFactory.create(),
                RawReportInfo(),
            )

.../tests/unit/test_upload_processing_task.py:757: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
tasks/upload_processor.py:381: in process_individual_report
    processing_result = report_service.build_report_from_raw_content(
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:888: in build_report_from_raw_content
    id=session_id_from_upload_id(upload.id),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

upload_id = None

    def session_id_from_upload_id(upload_id: int) -&gt; int:
        """
        Creates a unique `session_id` from an `upload_id`.
    
        The purpose of the `session_id` is to unique within one report.
        Turning the `upload_id` into the `session_id` trivially achieves this, as
        the `upload_id` is a unique database id.
    
        However in production, the `upload_id` exceeds 31 bits, which overflows
        the datatype of the `session_id`, which is a signed 32-bit integer
        (effectively 31-bits).
    
        We work around this problem by just masking off the high bits, essentially
        wrapping around at that number, and shifting a bit to avoid collisions
        with existing low session numbers.
        """
&gt;       return (upload_id &amp; (2**16 - 1)) &lt;&lt; 12
E       TypeError: unsupported operand type(s) for &amp;: 'NoneType' and 'int'

helpers/reports.py:39: TypeError

services.tests.test_report.TestReportService test_save_full_report

Stack Traces | 0.064s run time

self = &lt;worker.services.tests.test_report.TestReportService object at 0x7f2dd0f01640&gt;
dbsession = &lt;sqlalchemy.orm.session.Session object at 0x7f2dc48899d0&gt;
mock_storage = &lt;shared.storage.memory.MemoryStorageService object at 0x7f2dc46d02c0&gt;
sample_report = &lt;Report files=2&gt;
mock_configuration = &lt;shared.config.ConfigHelper object at 0x7f2dc487d400&gt;

    def test_save_full_report(
        self, dbsession, mock_storage, sample_report, mock_configuration
    ):
        mock_configuration.set_params(
            {
                "setup": {
                    "save_report_data_in_storage": {
                        "only_codecov": False,
                        "report_details_files_array": True,
                    },
                }
            }
        )
        commit = CommitFactory.create()
        dbsession.add(commit)
        dbsession.flush()
        current_report_row = CommitReport(commit_id=commit.id_)
        dbsession.add(current_report_row)
        dbsession.flush()
        report_details = ReportDetails(report_id=current_report_row.id_)
        dbsession.add(report_details)
        dbsession.flush()
        sample_report.sessions[0].archive = ".../to/upload/location"
        sample_report.sessions[
            0
        ].name = "this name contains more than 100 chars 1111111111111111111111111111111111111111111111111111111111111this is more than 100"
        report_service = ReportService({})
&gt;       res = report_service.save_full_report(commit, sample_report)

services/tests/test_report.py:3547: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12.../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:1186: in save_full_report
    self._attach_flags_to_upload(upload, session.flags if session.flags else [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = &lt;services.report.ReportService object at 0x7f2dc7f4d940&gt;
upload = &lt;database.models.reports.Upload object at 0x7f2dc44e87d0&gt;
flag_names = ['unit']

    def _attach_flags_to_upload(self, upload: Upload, flag_names: Sequence[str]):
        """Internal function that manages creating the proper `RepositoryFlag`s and attach the sessions to them
    
        Args:
            upload (Upload): Description
            flag_names (Sequence[str]): Description
    
        Returns:
            TYPE: Description
        """
        all_flags = []
        db_session = upload.get_db_session()
&gt;       repoid = upload.report.commit.repoid
E       AttributeError: 'NoneType' object has no attribute 'commit'

services/report/__init__.py:331: AttributeError

services.tests.test_report.TestReportService test_initialize_and_save_report_needs_backporting

Stack Traces | 0.069s run time

self = &lt;worker.services.tests.test_report.TestReportService object at 0x7f2dd10f5730&gt;
dbsession = &lt;sqlalchemy.orm.session.Session object at 0x7f2dc488d580&gt;
sample_commit_with_report_big = Commit&lt;659c950c1d6ea9da26848c0b2a9010906f68034d@repo&lt;582&gt;&gt;
mock_storage = &lt;shared.storage.memory.MemoryStorageService object at 0x7f2dc485ad80&gt;
mocker = &lt;pytest_mock.plugin.MockFixture object at 0x7f2dc48d17c0&gt;

    def test_initialize_and_save_report_needs_backporting(
        self, dbsession, sample_commit_with_report_big, mock_storage, mocker
    ):
        commit = sample_commit_with_report_big
        report_service = ReportService({})
        mocker.patch.object(
            ReportDetails, "_should_write_to_storage", return_value=True
        )
&gt;       r = report_service.initialize_and_save_report(commit)

services/tests/test_report.py:4240: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:268: in initialize_and_save_report
    self.save_full_report(commit, actual_report)
.../local/lib/python3.12....../site-packages/sentry_sdk/tracing_utils.py:673: in func_with_tracing
    return func(*args, **kwargs)
services/report/__init__.py:1186: in save_full_report
    self._attach_flags_to_upload(upload, session.flags if session.flags else [])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = &lt;services.report.ReportService object at 0x7f2dc48d3860&gt;
upload = &lt;database.models.reports.Upload object at 0x7f2dc46d2600&gt;
flag_names = []

    def _attach_flags_to_upload(self, upload: Upload, flag_names: Sequence[str]):
        """Internal function that manages creating the proper `RepositoryFlag`s and attach the sessions to them
    
        Args:
            upload (Upload): Description
            flag_names (Sequence[str]): Description
    
        Returns:
            TYPE: Description
        """
        all_flags = []
        db_session = upload.get_db_session()
&gt;       repoid = upload.report.commit.repoid
E       AttributeError: 'NoneType' object has no attribute 'commit'

services/report/__init__.py:331: AttributeError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

The problem here is that `order_number` is a 32-bit signed int (effectively 31-bit), and duplicating the `reports_upload.id` into it does not work, as that value already exceeds 31-bits in production. This tries to work around that problem by arranging that number to occupy bits 12-28.

Swatinem requested a review from a team September 18, 2024 12:14

Swatinem self-assigned this Sep 18, 2024

Swatinem marked this pull request as draft September 18, 2024 12:18

Swatinem added 3 commits September 18, 2024 17:10

Make session.id == upload.id (#713)

f91cede

fix various foreign key column definitions

cdf4bba

Swatinem force-pushed the swatinem/fix-sessionid-allocation branch from 1e40b96 to cdf4bba Compare September 18, 2024 15:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Derive the `session.id`/`order_number` from the `upload.id` #720

Derive the `session.id`/`order_number` from the `upload.id` #720

Swatinem commented Sep 18, 2024

codecov-staging bot commented Sep 18, 2024 •

edited by codecov-notifications bot

Loading

codecov-qa bot commented Sep 18, 2024 •

edited

Loading

codecov-public-qa bot commented Sep 18, 2024 •

edited

Loading

pytest

codecov bot commented Sep 18, 2024 •

edited

Loading

Derive the session.id/order_number from the upload.id #720

Are you sure you want to change the base?

Derive the session.id/order_number from the upload.id #720

Conversation

Swatinem commented Sep 18, 2024

codecov-staging bot commented Sep 18, 2024 • edited by codecov-notifications bot Loading

Codecov Report

codecov-qa bot commented Sep 18, 2024 • edited Loading

❌ 8 Tests Failed:

codecov-public-qa bot commented Sep 18, 2024 • edited Loading

❌ Failed Test Results:

pytest

codecov bot commented Sep 18, 2024 • edited Loading

❌ 8 Tests Failed:

Derive the `session.id`/`order_number` from the `upload.id` #720

Derive the `session.id`/`order_number` from the `upload.id` #720

codecov-staging bot commented Sep 18, 2024 •

edited by codecov-notifications bot

Loading

codecov-qa bot commented Sep 18, 2024 •

edited

Loading

codecov-public-qa bot commented Sep 18, 2024 •

edited

Loading

codecov bot commented Sep 18, 2024 •

edited

Loading