dataset lost harvest object when xml file timestamp changed without xml content change

[FuhuXia]

WAF source file timestamp changes without real content change is causing other issues such as #4425, but it also make dataset losing its harvest object on the UI, and potentially it is the biggest contributor to the db-solr-sync workload.

How to reproduce

Modify a XML file timestamp on a WAF souce, reharvest

Expected behavior

No change on the dataset. UI stays the same, no addition workload to db-solr-sync

Actual behavior

See the error in the fetch log

Document with GUID ### unchanged, skipping...

On the UI, dataset lost its harvest souce metadata info

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

[Jin-Sun-tts]

reproduce this issue in local test:

metadata update date were changed to new timestamp from the file, but lost its harvest souce metadata info

[Jin-Sun-tts]

After conducting local testing, identified the code responsible for the root cause of the issue: https://github.com/ckan/ckanext-spatial/blob/master/ckanext/spatial/harvesters/base.py#L709C1-L710C70.

In the testing, above rebuild index call did not refresh Solr from DB, and discovered that calling package_update or package_patch instead of package_index.index_package resolved the issue and displayed the metadata info in the UI.

will investigate alternative methods for rebuilding the index to address this problem.

[Jin-Sun-tts]

Tested the most recent version of Ckan, utilizing only two extensionsharvestand spatial without incorporating any additional customized modifications. was able to replicate the same problem.

[Jin-Sun-tts]

After conducting additional tests, it was observed that by adding model.Session.commit() prior to invoking package_index.index_package, the process functions as expected.

Furthermore, an examination of the package dictionary transmitted to Solr, same contents regardless of whether a commit was performed or not.

Further investigation is required to determine why package_index.index_package needs a database commit to refresh Solr.

[Jin-Sun-tts]

created upstream issue ckan/ckanext-spatial#324

[FuhuXia]

The root cause of the issue is illustrated in the above PR/commit. It shows the query with current=True is getting results with current=False. Could be a CKAN core bug that when querying updated records in an uncommited transaction, wrong results are returned.