Dependencies upgrade (#43)

* paramiko 3.4, bookops-bpl-solr & bookops-nypl-platfor 0.4, pytest 8.1.1, pytest-cov 5.0.0, pytest-mock 3.14.0, mypy 1.9, black 24.3 & moved mypy config to pyproject.toml * pymarc 5.1.2 & bookops-marc 0.10.0 * psycopg2 2.9.9 * version bump to 0.6.0 * bookops-worldcat 1.0.0 * pytest & coverage config moved to pyproject.toml * datetime.utcnow() > aware datetime.now() * updated changelog * typo fix in tool.poetry.group.dev.dependencies * checkout v4 & setup-python v5 * dependencies update * incorrectly applied timezone
BookOps-CAT · Mar 29, 2024 · ec5077b · ec5077b
1 parent 4fb5cf2
commit ec5077b
Show file tree

Hide file tree

Showing 29 changed files with 1,116 additions and 1,078 deletions.
diff --git a/.coveragerc b/.coveragerc
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
@@ -22,9 +22,9 @@ jobs:
         ports:
          - 5432:5432
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
       - name: Set up Python 3.9
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
           python-version: '3.9'
       - name: Install dependencies

diff --git a/README.md b/README.md
@@ -14,7 +14,7 @@ More info:
 [Sierra Scheduler configuration](https://github.com/BookOps-CAT/NightShift/blob/main/docs/sierra.md) | [WorldCat record matching](https://github.com/BookOps-CAT/NightShift/blob/main/docs/matching.md) | [Record manipulation](https://github.com/BookOps-CAT/NightShift/blob/main/docs/manipulating.md) | [Sierra loading instructions](https://github.com/BookOps-CAT/NightShift/blob/main/docs/loading.md) 
 
 ## Version
-> 0.5.0
+> 0.6.0
 
 ## Local Installation & Usage
 
@@ -61,6 +61,25 @@ If a good match in WorldCat has been found, the bot manipulates the downloaded f
 If for any reason the execution of the routine is interrupted (API error, etc.), the process can be restarted using `run [local, prod]` command again. The bot will pick up exactly where it left.
 
 ## Changelog
+[0.6.0] - 2024-03-28
+### Changed
++ updated dependencies:
+  + pymarc (5.1.2)
+  + psycopg2 (2.9.9)
+  + bookops-worldcat (1.0.0)
+  + paramiko (3.4.0)
+  + bookops-bpl-solr (0.4.0)
+  + bookops-nypl-platform (0.4.0)
+  + bookops-marc (0.10.0)
++ updated dev dependencies:
+  + pytest (8.1.1)
+  + black (24.3.0)
+  + pytest-cov (5.0.0)
+  + pytest-mock (3.14.0)
+  + mypy (1.9.0)
++ coverage & pytest configuration moved to `pyproject.toml`
++ `datetime.utcnow` changed to aware `datetime.now` in anticipation of deprecation of the former
++ refactored MARC fields manipulation to reflect a new way to construct subfields that uses pymarc's `Subfields`
 [0.5.0] - 2023-04-12
 ### Changed
 + "Electronic books" and "Electronic audiobooks" actively removed from Worldcat records
@@ -113,4 +132,5 @@ If for any reason the execution of the routine is interrupted (API error, etc.),
 [0.2.0]: https://github.com/BookOps-CAT/NightShift/compare/0.1.0...0.2.0
 [0.3.0]: https://github.com/BookOps-CAT/NightShift/compare/0.2.0...0.3.0
 [0.4.0]: https://github.com/BookOps-CAT/NightShift/compare/0.3.0...0.4.0
-[0.5.0]: https://github.com/BookOps-CAT/NightShift/compare/0.4.0...0.5.0
+[0.5.0]: https://github.com/BookOps-CAT/NightShift/compare/0.4.0...0.5.0
+[0.6.0]: https://github.com/BookOps-CAT/NightShift/compare/0.5.0...0.6.0
diff --git a/dev-requirements.txt b/dev-requirements.txt
@@ -1,45 +1,43 @@
-attrs==22.2.0; python_version >= "3.7"
-bcrypt==4.0.1; python_version >= "3.6"
-black==22.12.0; python_version >= "3.7"
-bookops-bpl-solr @ git+https://github.com/BookOps-CAT/[email protected] ; python_version >= "3.7" and python_version < "4.0"
-bookops-marc @ git+https://github.com/BookOps-CAT/[email protected] ; python_version >= "3.7" and python_version < "4.0"
-bookops-nypl-platform @ git+https://github.com/BookOps-CAT/[email protected] ; python_version >= "3.7" and python_version < "4.0"
-bookops-worldcat==0.5.0; python_version >= "3.7" and python_version < "4.0"
-certifi==2022.12.7; python_version >= "3.7" and python_version < "4"
-cffi==1.15.1; python_version >= "3.6"
-charset-normalizer==3.1.0; python_version >= "3.7" and python_version < "4" and python_full_version >= "3.7.0"
-click==8.1.3; python_version >= "3.7"
-colorama==0.4.6; python_version >= "3.7" and python_full_version < "3.0.0" and sys_platform == "win32" and platform_system == "Windows" or sys_platform == "win32" and python_version >= "3.7" and python_full_version >= "3.7.0" and platform_system == "Windows"
-coverage==6.5.0; python_version >= "3.7"
-cryptography==40.0.1; python_version >= "3.6"
-exceptiongroup==1.1.1; python_version < "3.11" and python_version >= "3.7"
-greenlet==2.0.2; python_version >= "3" and python_full_version < "3.0.0" and (platform_machine == "aarch64" or platform_machine == "ppc64le" or platform_machine == "x86_64" or platform_machine == "amd64" or platform_machine == "AMD64" or platform_machine == "win32" or platform_machine == "WIN32") and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.6.0") or python_version >= "3" and (platform_machine == "aarch64" or platform_machine == "ppc64le" or platform_machine == "x86_64" or platform_machine == "amd64" or platform_machine == "AMD64" or platform_machine == "win32" or platform_machine == "WIN32") and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.6.0") and python_full_version >= "3.5.0"
-idna==3.4; python_version >= "3.7" and python_version < "4"
-iniconfig==2.0.0; python_version >= "3.7"
-loggly-python-handler==1.0.1
-mypy-extensions==1.0.0; python_version >= "3.7"
-mypy==0.971; python_version >= "3.6"
-packaging==23.0; python_version >= "3.7"
-paramiko==2.12.0
-pathspec==0.11.1; python_version >= "3.7"
-platformdirs==3.2.0; python_version >= "3.7"
-pluggy==1.0.0; python_version >= "3.7"
-psycopg2==2.9.6; python_version >= "3.6"
-pycparser==2.21; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-pymarc==4.2.2; python_version >= "3.6"
-pynacl==1.5.0; python_version >= "3.6"
-pytest-cov==3.0.0; python_version >= "3.6"
-pytest-mock==3.10.0; python_version >= "3.7"
-pytest-sftpserver==1.3.0
-pytest==7.2.2; python_version >= "3.7"
-pyyaml==6.0; python_version >= "3.6"
-requests-futures==1.0.0
-requests==2.28.2; python_version >= "3.7" and python_version < "4"
-six==1.16.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0"
-sqlalchemy==1.4.47; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
-tomli==2.0.1; python_version < "3.11" and python_version >= "3.7" and python_full_version <= "3.11.0a6"
-types-paramiko==2.12.0.3
-types-requests==2.28.11.17
-types-urllib3==1.26.25.10
-typing-extensions==4.5.0; python_version < "3.10" and python_version >= "3.7"
-urllib3==1.26.15; python_version >= "3.7" and python_full_version < "3.0.0" and python_version < "4" or python_version >= "3.7" and python_version < "4" and python_full_version >= "3.6.0"
+bcrypt==4.1.2 ; python_version >= "3.9" and python_version < "4.0"
+black==24.3.0 ; python_version >= "3.9" and python_version < "4.0"
+bookops-bpl-solr @ git+https://github.com/BookOps-CAT/bookops-bpl-solr@8fa8ddef6338090d2a71e1779f4dae8264cdcc90 ; python_version >= "3.9" and python_version < "4.0"
+bookops-marc @ git+https://github.com/BookOps-CAT/bookops-marc.git@cd89f82f2dd093907e04871b1ba40b3f0407a9f4 ; python_version >= "3.9" and python_version < "4.0"
+bookops-nypl-platform @ git+https://github.com/BookOps-CAT/bookops-nypl-platform@711229b9f01760e52f3d7faad05965ab5b83ad53 ; python_version >= "3.9" and python_version < "4.0"
+bookops-worldcat==1.0.0 ; python_version >= "3.9" and python_version < "4.0"
+certifi==2024.2.2 ; python_version >= "3.9" and python_version < "4.0"
+cffi==1.16.0 ; python_version >= "3.9" and python_version < "4.0"
+charset-normalizer==3.3.2 ; python_version >= "3.9" and python_version < "4.0"
+click==8.1.7 ; python_version >= "3.9" and python_version < "4.0"
+colorama==0.4.6 ; python_version >= "3.9" and python_version < "4.0" and (sys_platform == "win32" or platform_system == "Windows")
+coverage[toml]==6.5.0 ; python_version >= "3.9" and python_version < "4.0"
+cryptography==42.0.5 ; python_version >= "3.9" and python_version < "4.0"
+exceptiongroup==1.2.0 ; python_version >= "3.9" and python_version < "3.11"
+greenlet==3.0.3 ; python_version >= "3.9" and (platform_machine == "aarch64" or platform_machine == "ppc64le" or platform_machine == "x86_64" or platform_machine == "amd64" or platform_machine == "AMD64" or platform_machine == "win32" or platform_machine == "WIN32") and python_version < "4.0"
+idna==3.6 ; python_version >= "3.9" and python_version < "4.0"
+iniconfig==2.0.0 ; python_version >= "3.9" and python_version < "4.0"
+loggly-python-handler==1.0.1 ; python_version >= "3.9" and python_version < "4.0"
+mypy-extensions==1.0.0 ; python_version >= "3.9" and python_version < "4.0"
+mypy==1.9.0 ; python_version >= "3.9" and python_version < "4.0"
+packaging==24.0 ; python_version >= "3.9" and python_version < "4.0"
+paramiko==3.4.0 ; python_version >= "3.9" and python_version < "4.0"
+pathspec==0.12.1 ; python_version >= "3.9" and python_version < "4.0"
+platformdirs==4.2.0 ; python_version >= "3.9" and python_version < "4.0"
+pluggy==1.4.0 ; python_version >= "3.9" and python_version < "4.0"
+psycopg2==2.9.9 ; python_version >= "3.9" and python_version < "4.0"
+pycparser==2.21 ; python_version >= "3.9" and python_version < "4.0"
+pymarc==5.1.2 ; python_version >= "3.9" and python_version < "4.0"
+pynacl==1.5.0 ; python_version >= "3.9" and python_version < "4.0"
+pytest-cov==5.0.0 ; python_version >= "3.9" and python_version < "4.0"
+pytest-mock==3.14.0 ; python_version >= "3.9" and python_version < "4.0"
+pytest-sftpserver==1.3.0 ; python_version >= "3.9" and python_version < "4.0"
+pytest==8.1.1 ; python_version >= "3.9" and python_version < "4.0"
+pyyaml==6.0.1 ; python_version >= "3.9" and python_version < "4.0"
+requests-futures==1.0.1 ; python_version >= "3.9" and python_version < "4.0"
+requests==2.31.0 ; python_version >= "3.9" and python_version < "4.0"
+six==1.16.0 ; python_version >= "3.9" and python_version < "4.0"
+sqlalchemy==1.4.52 ; python_version >= "3.9" and python_version < "4.0"
+tomli==2.0.1 ; python_version >= "3.9" and python_full_version <= "3.11.0a6"
+types-paramiko==3.4.0.20240311 ; python_version >= "3.9" and python_version < "4.0"
+types-requests==2.31.0.20240311 ; python_version >= "3.9" and python_version < "4.0"
+typing-extensions==4.10.0 ; python_version >= "3.9" and python_version < "4.0"
+urllib3==2.2.1 ; python_version >= "3.9" and python_version < "4.0"
diff --git a/mypy.ini b/mypy.ini
diff --git a/nightshift/__init__.py b/nightshift/__init__.py
@@ -1,2 +1,2 @@
-__version__ = "0.5.0"
+__version__ = "0.6.0"
 __title__ = "NightShift"
diff --git a/nightshift/comms/worldcat.py b/nightshift/comms/worldcat.py
@@ -11,7 +11,7 @@
 from bookops_worldcat import WorldcatAccessToken, MetadataSession
 from bookops_worldcat.errors import (
     WorldcatAuthorizationError,
-    WorldcatSessionError,
+    WorldcatRequestError,
 )
 from requests import Response
 
@@ -147,8 +147,6 @@ def _get_credentials(self) -> dict:
             key=os.getenv(f"WC{self.library}_KEY"),
             secret=os.getenv(f"WC{self.library}_SECRET"),
             scopes="WorldCatMetadataAPI",
-            principal_id=os.getenv(f"WC{self.library}_PRINCIPALID"),
-            principal_idns=os.getenv(f"WC{self.library}_PRINCIPALIDNS"),
             agent=f"{__title__}/{__version__}",
         )
 
@@ -264,7 +262,7 @@ def get_brief_bibs(
                     continue
 
                 for payload in payloads:
-                    response = self.session.search_brief_bibs(
+                    response = self.session.brief_bibs_search(
                         **payload,
                         inCatalogLanguage="eng",
                         orderBy="mostWidelyHeld",
@@ -290,8 +288,8 @@ def get_brief_bibs(
 
                 yield (resource, brief_bib_response)
 
-        except WorldcatSessionError:
-            logger.error(f"WorldcatSessionError. Aborting.")
+        except WorldcatRequestError:
+            logger.error(f"WorldcatRequestError. Aborting.")
             raise
 
     def get_full_bibs(
@@ -302,14 +300,12 @@ def get_full_bibs(
         """
         try:
             for resource in resources:
-                response = self.session.get_full_bib(
-                    oclcNumber=resource.oclcMatchNumber
-                )
+                response = self.session.bib_get(oclcNumber=resource.oclcMatchNumber)
                 logger.debug(
                     f"Full bib Worldcat request for {self.library} Sierra bib # "
                     f"b{resource.sierraId}a: {response.url}."
                 )
                 yield (resource, response.content)
-        except WorldcatSessionError:
-            logger.error("WorldcatSessionError. Aborting.")
+        except WorldcatRequestError:
+            logger.error("WorldcatRequestError. Aborting.")
             raise
diff --git a/nightshift/config/config.yaml.example b/nightshift/config/config.yaml.example
@@ -6,12 +6,8 @@ POSTGRES_PORT: postgres_port
 POSTGRES_DB: postgred_db_name
 WCNYP_KEY: nypl_worldcat_key
 WCNYP_SECRET: nypl_worldcat_secret
-WCNYP_PRINCIPALID: nypl_principal_id
-WCNYP_PRINCIPALIDNS: nypl_principal_idns
 WCBPL_KEY: bpl_worldcat_key
 WCBPL_SECRET: bpl_worlcat_secret
-WCBPL_PRINCIPALID: bpl_worlcat_principal_id
-WCBPL_PRINCIPALIDNS: bpl_worlcat_principal_idns
 LOGGLY_TOKEN: loggly_token
 LOG_HANDLERS: "loggly,file,console"
 SFTP_HOST: sftp_host

diff --git a/nightshift/datastore.py b/nightshift/datastore.py
@@ -4,7 +4,7 @@
 NightShift's database schema.
 """
 from contextlib import contextmanager
-from datetime import datetime
+from datetime import datetime, timezone
 import os
 
 from sqlalchemy import (
@@ -98,7 +98,7 @@ class Event(Base):
     __tablename__ = "event"
 
     nid = Column(Integer, primary_key=True)
-    timestamp = Column(DateTime, nullable=False, default=datetime.utcnow())
+    timestamp = Column(DateTime, nullable=False, default=datetime.now(timezone.utc))
     libraryId = Column(Integer, ForeignKey("library.nid"), nullable=False)
     sierraId = Column(Integer, nullable=False)
     bibDate = Column(Date, nullable=False)
@@ -142,7 +142,7 @@ class OutputFile(Base):
     nid = Column(Integer, primary_key=True)
     libraryId = Column(Integer, ForeignKey("library.nid"), nullable=False)
     handle = Column(String, nullable=False)
-    timestamp = Column(DateTime, default=datetime.utcnow())
+    timestamp = Column(DateTime, default=datetime.now(timezone.utc))
 
     def __repr__(self):
         return (
@@ -291,7 +291,7 @@ class SourceFile(Base):
     nid = Column(Integer, primary_key=True)
     libraryId = Column(Integer, ForeignKey("library.nid"), nullable=False)
     handle = Column(String, nullable=False)
-    timestamp = Column(DateTime, default=datetime.utcnow())
+    timestamp = Column(DateTime, default=datetime.now(timezone.utc))
 
     def __repr__(self):
         return (
@@ -315,7 +315,7 @@ class WorldcatQuery(Base):
     )
     match = Column(Boolean, nullable=False)
     response = Column(JSONB)
-    timestamp = Column(DateTime, default=datetime.utcnow(), nullable=False)
+    timestamp = Column(DateTime, default=datetime.now(timezone.utc), nullable=False)
 
     def __repr__(self):
         return (

diff --git a/nightshift/datastore_transactions.py b/nightshift/datastore_transactions.py
@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 from collections import namedtuple
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
 from typing import Optional
 
 from sqlalchemy import and_, create_engine, delete, func, inspect, update
@@ -156,7 +156,7 @@ def add_event(session: Session, resource: Resource, status: str) -> Event:
         bibDate=resource.bibDate,
         resourceCategoryId=resource.resourceCategoryId,
         status=status,
-        timestamp=datetime.utcnow(),
+        timestamp=datetime.now(timezone.utc),
     )
 
     session.add(instance)
@@ -241,7 +241,7 @@ def delete_resources(session: Session, resourceCategoryId: int, age: int) -> int
         session.query(Resource)
         .filter(
             Resource.resourceCategoryId == resourceCategoryId,
-            Resource.bibDate < datetime.utcnow() - timedelta(days=age),
+            Resource.bibDate < datetime.now(timezone.utc) - timedelta(days=age),
         )
         .delete()
     )
@@ -354,7 +354,7 @@ def retrieve_expired_resources(
             Resource.resourceCategoryId == resourceCategoryId,
             Resource.status == "open",
             Resource.bibDate
-            < datetime.utcnow().date() - timedelta(days=expiration_age),
+            < datetime.now(timezone.utc).date() - timedelta(days=expiration_age),
         )
         .all()
     )
@@ -411,8 +411,8 @@ def retrieve_open_older_resources(
             Resource.resourceCategoryId == resourceCategoryId,
             Resource.status == "open",
             Resource.oclcMatchNumber == None,
-            Resource.bibDate > datetime.utcnow() - timedelta(days=maxAge),
-            Resource.bibDate < datetime.utcnow() - timedelta(days=minAge),
+            Resource.bibDate > datetime.now(timezone.utc) - timedelta(days=maxAge),
+            Resource.bibDate < datetime.now(timezone.utc) - timedelta(days=minAge),
         )
         .group_by(Resource.nid)
         .subquery()
@@ -550,7 +550,7 @@ def set_resources_to_expired(
         .filter(
             Resource.resourceCategoryId == resourceCategoryId,
             Resource.status == "open",
-            Resource.bibDate < datetime.utcnow().date() - timedelta(days=age),
+            Resource.bibDate < datetime.now(timezone.utc).date() - timedelta(days=age),
         )
         .update({"status": "expired"})
     )

diff --git a/nightshift/marc/marc_parser.py b/nightshift/marc/marc_parser.py
@@ -178,16 +178,16 @@ def _map_data(self, bib: Bib, resource_category: str) -> Resource:
         sierraId = bib.sierra_bib_id_normalized()
         resourceCategoryId = self._res_cat[resource_category].nid
         bibDate = bib.created_date()
-        author = bib.author()
-        title = bib.title()
-        pubDate = bib.pubyear()
+        author = bib.author
+        title = bib.title
+        pubDate = bib.pubyear
         congressNumber = bib.lccn()
         controlNumber = bib.control_number()
         distributorNumber = bib.overdrive_number()
         suppressed = bib.suppressed()
         otherNumber = bib.upc_number()
         srcFieldsToKeep = self._fields2keep(bib, resource_category)
-        standardNumber = bib.isbn()
+        standardNumber = bib.isbn
 
         resource = Resource(
             sierraId=sierraId,