download.py failing with "Connection reset by peer"

[pdurbin]

I'm on 1d57672 and tried to run python3 metrics.py with the list of Dataverse installations in config.json.sample but got the following error and stack trace. We need better error handling, obviously. 😄

$ python3 metrics.py 
Traceback (most recent call last):
  File "/usr/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "metrics.py", line 9, in <module>
    main()
  File "metrics.py", line 5, in main
    download.main()
  File "/var/www/html/tmp/dataverse-metrics/download.py", line 31, in main
    process_single_endpoints(installation, single_endpoints, api_response_cache_dir)
  File "/var/www/html/tmp/dataverse-metrics/download.py", line 84, in process_single_endpoints
    process_single_endpoint(installation, endpoint, api_response_cache_dir)
  File "/var/www/html/tmp/dataverse-metrics/download.py", line 89, in process_single_endpoint
    response = urlrequest.urlopen(url)
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 104] Connection reset by peer>

[pdurbin]

@donsizemore I know you'd like some better error handling and I wanted to mention that in pull request #21 I added some error handling for the new "show the number of contributors to GitHub repos" code. I'm sending errors to stderr like you asked and I'd be happy for you to try it out. To test it I've been changing https://api.github.com to some server I control to produce 404s, if that makes sense. I have not applied it to other parts of the code (downloading metrics from Dataverse installations) because I'd rather not put all of that through QA right now.

[pdurbin]

In the line below I talk about the difficulties I've had in implementing error handling while also having no dependencies (batteries included) and Python 2 and Python 3 compatibility:

https://github.com/IQSS/dataverse-metrics/blob/v0.2.5/download.py#L114

I think we should consider dropping Python 2 support before we tackle this issue of better error handling.