Server: Corpus did not restart

[david-allison]

Watchtower:

time="2024-05-08T15:06:15Z" level=info msg="Watchtower 1.7.1"
time="2024-05-08T15:06:15Z" level=info msg="Using no notifications"
time="2024-05-08T15:06:15Z" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2024-05-08T15:06:15Z" level=info msg="Scheduling first run: 2024-05-09 04:00:00 +0000 UTC"
time="2024-05-08T15:06:15Z" level=info msg="Note that the first check will be performed in 12 hours, 53 minutes, 44 seconds"
time="2024-05-09T04:00:10Z" level=info msg="Found new ghcr.io/david-allison/manx-corpus-search:master image (89a7eb2953bf)"
time="2024-05-09T04:00:10Z" level=info msg="Stopping /corpus-search-1 (47491c254ebc) with SIGTERM"
time="2024-05-09T04:00:20Z" level=info msg="Creating /corpus-search-1"
time="2024-05-09T04:00:21Z" level=info msg="Removing image fec54ce12869"
time="2024-05-09T04:00:21Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=1 notify=no
time="2024-05-10T04:00:02Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-11T04:00:08Z" level=info msg="Found new ghcr.io/david-allison/manx-corpus-search:master image (4d8490dc7036)"
time="2024-05-11T04:00:09Z" level=info msg="Stopping /corpus-search-1 (e90e4b27c188) with SIGTERM"
time="2024-05-11T04:00:13Z" level=info msg="Creating /corpus-search-1"
time="2024-05-11T04:00:14Z" level=info msg="Removing image 89a7eb2953bf"
time="2024-05-11T04:00:14Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=1 notify=no
time="2024-05-12T04:00:08Z" level=info msg="Found new ghcr.io/david-allison/manx-corpus-search:master image (fdbe884f96ad)"
time="2024-05-12T04:00:09Z" level=info msg="Stopping /corpus-search-1 (b84e75d567f9) with SIGTERM"
time="2024-05-12T04:00:12Z" level=info msg="Creating /corpus-search-1"
time="2024-05-12T04:00:14Z" level=info msg="Removing image 4d8490dc7036"
time="2024-05-12T04:00:14Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=1 notify=no
time="2024-05-13T04:00:09Z" level=info msg="Found new ghcr.io/david-allison/manx-corpus-search:master image (4725681e5b21)"
time="2024-05-13T04:00:10Z" level=info msg="Stopping /corpus-search-1 (342863e0f787) with SIGTERM"
time="2024-05-13T04:00:20Z" level=info msg="Creating /corpus-search-1"
time="2024-05-13T04:00:21Z" level=info msg="Removing image fdbe884f96ad"
time="2024-05-13T04:00:21Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=1 notify=no
time="2024-05-14T04:00:02Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-15T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-16T04:00:03Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-17T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-18T04:00:02Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-19T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-20T04:00:02Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-21T04:00:02Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-22T04:00:02Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-23T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-24T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-25T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-26T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-27T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-28T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-29T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-30T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-05-31T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-06-01T04:00:02Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-06-02T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-06-03T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-06-04T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-06-05T04:00:02Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
time="2024-06-06T04:00:01Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no

• no recent updates

docker logs corpus-search-1

Downloading Cregeen...
Downloading Kelly Manx to English...
Copying dictionaries to /Resources
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
      Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning
Failed to init anonymous analytics. Was CORPUS_SEARCH_SEGMENT_KEY set?
Loaded 16 documents
Loaded 710 documents
compacted in 19134
1855552 in 726 documents
Found 8 latest documents
warn: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[16]
      The WebRootPath was not found: /app/wwwroot. Static files may be unavailable.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:8080
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "06/06/2024 15:23:58 +00:00", the heartbeat has been running for "00:00:01.2456382" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "06/06/2024 15:24:00 +00:00", the heartbeat has been running for "00:00:01.1201565" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "06/06/2024 15:25:13 +00:00", the heartbeat has been running for "00:00:03.1679506" which is longer than "00:00:01". This could be caused by thread pool starvation.
warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "06/06/2024 15:25:30 +00:00", the heartbeat has been running for "00:00:05.3241903" which is longer than "00:00:01". This could be caused by thread pool starvation.
./tools/init.sh: line 37:    99 Killed                  dotnet CorpusSearch.dll

[david-allison]

root@ubuntu-s-1vcpu-1gb-lon1-01:~# dmesg -T | egrep -i 'killed process'
[Thu Jun  6 15:04:24 2024] Out of memory: Killed process 1963353 (dotnet) total-vm:273701724kB, anon-rss:1307956kB, file-rss:0kB, shmem-rss:44392kB, UID:0 pgtables:3244kB oom_score_adj:0

[david-allison]

killed corpus-2 as it was no longer needed

sudo systemctl stop manx-corpus.service