Large dataset issues

[sethmenghi]

When downloading large datasets, there are issues with retrieving the data. Using an AsyncClient solved this issue as the dataretrieval package failed. Here is what was used to fix this issue:

import httpx

import json_stream.httpx

async def get_dataframe(
        url: str,
        location_id: str
    ):):      
      async with httpx.AsyncClient() as client:
            response = await self.get_data(url, request_params, client)
            response.raise_for_status()

            df= self.parse_data(response, merged_df)
            ...

    async def get_data(url, request_params: dict, client):
        """Tenacity retry api calls."""

        _retry = retry(stop=stop_after_attempt(7))
        return await _retry(client.post)(
            url,
            json=request_params,
            follow_redirects=True,
            timeout=None,
        )