Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the null terminator from flask_utils.py #709

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

NEC-Vishal
Copy link
Contributor

Proposed changes

Types of changes

What types of changes does your code introduce to the project?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Maintenance (update of libraries or other dependences)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have read the CONTRIBUTING doc
  • I have signed the CLA
  • I have added tests that prove my fix is effective or that my feature works
  • I have run all the existing tests locally (not just those related to my feature) and there are no errors
  • After the last push to the PR branch, I have run the lint script locally and there are no changes to the code base
  • I have updated the RELEASE NOTES
  • I have added necessary documentation (if appropriate)
  • Any dependent changes have been merged and published in downstream modules

Further comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions
Copy link
Contributor

github-actions bot commented Nov 28, 2022

CLA Assistant Lite bot All contributors have signed the CLA ✍️

@NEC-Vishal
Copy link
Contributor Author

#707

Copy link
Member

@c0c0n3 c0c0n3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NEC-Vishal we're not there yet... :-)

# TODO how to get rid of the null terminator in an efficient and **simple**
# way? I could use the same put-back approach as in itersplit but I'd rather
# keep it simple.
yield '\n]'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NEC-Vishal nice try, but it won't work :-)
In fact, if the input iterable isn't empty the output JSON array will contain an extra comma before the array end symbol, e.g. [1, 2, 3] ~~~> "[\n1,\n2,\n3,\n]" which is not a valid JSON array!

@c0c0n3
Copy link
Member

c0c0n3 commented Jan 6, 2023

@NEC-Vishal before going ahead w/ this PR, can you please have a read through #707. I realised the TODO in the code isn't really helpful, it's so vague that's basically impossible to understand. #707 now explains the whole thing in detail.

@c0c0n3
Copy link
Member

c0c0n3 commented Jan 6, 2023

@NEC-Vishal another thing I realised is that if we merged the small change you implemented, the whole WQ query API would break. This is my fault b/c I didn't realise we needed to have test cases to cover the array streaming format. So let's add test cases. Can please create a Python package wq.ql.tests and a file test_flaskutils.py with the following content

import json
import pytest

from wq.ql.flaskutils import *


class Item(BaseModel):
    id: int

    @staticmethod
    def range(stop: int) -> ['Item']:
        return [Item(id=x) for x in range(stop)]


items_array_supply = [
    Item.range(stop) for stop in range(4)
    # [], [Item(id=1)], [Item(1), Item(2)], ...
]


def stream_to_json_doc(xs: Iterable[BaseModel]) -> str:
    lines = [line for line in json_array_streamer(xs)]
    return "".join(lines)


def extract_item_ids(streamed_doc: str) -> [int]:
    item_array = json.loads(streamed_doc)
    return [x['id'] for x in item_array if x is not None]


@pytest.mark.parametrize('items', items_array_supply)
def test_json_array_streamer(items: [Item]):
    want = [x.id for x in items]
    got = extract_item_ids(stream_to_json_doc(items))

    assert want == got

@c0c0n3
Copy link
Member

c0c0n3 commented Jan 6, 2023

@NEC-Vishal then you should hook up the new tests to the ql suite. If you run the tests in that file against your code that zaps the JSON null value, you should see a JSON decode exception.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants