v2

README.md

@@ -1,14 +1,14 @@
 # DeepPhe Stream Multi-Container Stack - release branch
 
-This multi-container docker stack consists of the following 2 containers: 
+This multi-container docker stack consists of the following 2 contaienrs: 
 
-- 1: `dphe-stream-nginx` (reverse proxy and load balancer)
-- 2: `dphe-stream` (document and patient summary REST API)
+- 1 : `dphe-stream-nginx` (reverse proxy)
+- 2 : `dphe-stream` (document and patient summary REST API)
 
 ## Changes needed to make a release
 
-- 1: Place the source code zip file, for instance `v0.1.0-cr-release.zip`, to the directory `dphe-stream`. This zip file will be used to build the `dphe-stream` docker image. 
-- 2: Edit the `dphe-stream/Dockerfile` and specify to use the target release version, for instance `0.1.0`. 
+- 1: Place the source code zip file, for instance `v0.2.0-cr-release.zip`, to the directory `dphe-stream`. And this zip file will be used to build the `dphe-stream` docker image. 
+- 2: Edit the `dphe-stream/Dockerfile` and specify to use the target release version, for instance `0.2.0`. 
 
 ## Overview of tools
 
@@ -17,9 +17,9 @@ This multi-container docker stack consists of the following 2 containers:
 
 Note: Docker Compose requires Docker to be installed and running first.
 
-## Docker post-installation configurations
+### Docker post-installation configurations
 
-The Docker daemon binds to a Unix socket instead of a TCP port. By default, that Unix socket is owned by the user root and other users can only access it using sudo. The Docker daemon always runs as the root user. 
+The Docker daemon binds to a Unix socket instead of a TCP port. By default that Unix socket is owned by the user root and other users can only access it using sudo. The Docker daemon always runs as the root user. 
 
 If you're using Linux and you don't want to preface the docker command with sudo, you can add users to the `docker` group:
 
@@ -31,64 +31,71 @@ The log out and log back in so that your group membership is re-evaluated. If te
 
 Note: the following instructions with docker commands are based on managing Docker as a non-root user.
 
-## Docker build
+## Build docker images
 
-### Specify authentication token
+### Specify auth token
 
-Before starting building the child images, specify the auth token in `dphe-stream/deepphe.properties`. This auth token will be used later when interacting with the REST API calls via the standard HTTP request `Authorization` header with the Bearer scheme in the form of:
+Before starting building the child images, specify the auth token in `dphe-stream/deepphe.properties`. This auth token will be used later when interacting with the REST API calls via the standard HTTP request `Authorization` header with the Bearer scheme:
 
 ````
 Authorization: Bearer <token>
 ````
 
-This auth layer as a gateway will apply to all the HTTP requests before they can reach to the actual API endpoints.
+This auth layer applies to all the REST API requests before they can reach to the actual API endpoints.
 
-A default token is provided in the `dphe-stream/deepphe.propertiers` file. For greater security, it is strongly advised that you change this token to a new value for any deployed instance.
+A default token is provided in the `dphe-stream/deepphe.propertiers` file. For greater security, it is strongly advised that you change this token to a new value for deployment instance.
 
-### Build docker images
+### Build `dphe-stream-nginx` and `dphe-stream` images
 
-Under the project root directory where we can find the `docker-compose.yml`:
+Next go back to the project root directory where you can find the `docker-compose.yml`:
 
 ````
-docker-compose build
+docker-compose build --no-cache
 ````
 
-#### Start up services
+### Vulnerability scanning for local images
+
+Vulnerability scanning for Docker local images allows us to review the security state of the container images and take actions to fix issues identified during the scan, resulting in more secure deployments. The `scan` command is available by default in Docker version 20.10.x and newer.
+
+```
+docker scan --dependency-tree --file ./dphe-stream/Dockerfile dphe-stream:0.2.0-cr
+docker scan --dependency-tree --file ./dphe-stream-nginx/Dockerfile dphe-stream-nginx:0.2.0-cr
+```
+
+## Start up services
 
 There are two configurable environment variables to keep in mind before starting up the containers:
 
 - `HOST_UID`: the user id on the host machine to be mapped to all the containers. Default to 1000 if not set or null.
 - `HOST_GID`: the user's group id on the host machine to be mapped to all the containers. Default to 1000 if not set or null.
 
-We can set and verify the environment variable like below:
+We can use the default values if the ouput of the below command is 1000 for both `uid` and `gid` of the current user who's going to spin up the containers.
 
 ````
-export HOST_UID=1000
-echo $HOST_UID
-export HOST_GID=1000
-echo $HOST_GID
+id
 ````
 
-In security practice, the processes within a running container should not run as root, or assume that they are root. The system user on the host machine should be in the docker group, and it should also be the user who builds the images and starts the containers. That's why we wanted to use this user's UID and GID within the containers to avoid security holes and file system permission issues as well.
+In security practice, the processes within a running container should not run as root, or assume that they are root. The system user on the host machine should be in the docker group and it should also be the user who builds the images and starts the containers. That's why we wanted to use this user's UID and GID within the containers to avoid security holes and file system permission issues as well.
 
 ````
 docker-compose up -d
 ````
 
-This command spins up all the services (in the background and leaves them running) defined in the `docker-compose.yml` and aggregates the output of each container. 
+This command spins up all the services (in the background as detached mode and leaves them running) defiened in the `docker-compose.yml` and aggregates the output of each container. Make sure the port `8080` and `8181` are not already allocated, otherwise the containers would fail to start.
 
 Note: the initialization of containers takes some time, you can use the following command in another terminal window to monitor the progress:
 
 ````
 docker-compose logs -f --tail="all" 
 ````
 
+
+## Interact with the DeepPhe REST API
+
 You will have the following API base URL for the REST API container:
 
 - `dphe-stream`: `http://localhost:8080/deepphe`
 
-## Interact with the REST API
-
 Please remember that you'll need to send over the auth token (specified prior the docker build) in the `Authorization` header for each HTTP request:
 
 ````
@@ -165,6 +172,8 @@ curl -i -X GET http://localhost:8080/deepphe/summarizePatient/patient/patientX \
      -H "Authorization: Bearer AbCdEf123456"
 ````
 
+## Manage the contaners
+
 ### Shell into the running container
 
 Sometimes you may want to shell into a running container to check more details, this can be done by:
@@ -197,14 +206,47 @@ This command stops both containers of this project and removes them as well the
 Note: At this time DeepPhe Stream could be run with a single container.  The multi-container stack exists to facilitate addition future workflows that may require additional containers.   
 
 
-## Run integration tests manually
+## Integration tests
 
-Once the containers are up running, we can run some integration tests written in Python within the `dphe-stream-nginx` container manually to verify the pipeline output by submitting some sample reports to the REST API. 
+Once the containers are up running, we can execute some integration tests written in Python to verify the pipeline output by submitting some sample reports to the REST API. The tests will be executed against the `dphe-stream-nginx` container, which proxies the requests to the backend REST API service.
 
-The test cases and configuration are located at `dphe-stream-dock/dphe-stream-nginx/integration-test`. If a different auth token is specified during the image creation phase, that same auth token should be specified in the `test.cfg` as well.
+The test cases and configuration are located at `dphe-stream-dock/dphe-stream-nginx/integration-test`. If a different auth token is specified during the image creation phase, that same auth token should be specified int he `test.cfg` as well.
+
+### Run the tests manually within the container
 
 ````
 docker exec -it dphe-stream-nginx bash
 cd integration-test/
 python3 test.py
 ````
+
+### Add more tests and run against the container
+
+We can also add more test cases within the `dphe-stream-nginx/integration-test` directory to improve the coverage and run the tests against the running contianers. To do so, we can create a Python virtual environment and install the dependencies:
+
+```
+python3 -m venv venv
+source venv/bin/activate
+pip install -r requirements.txt
+```
+
+Once the test changes are made, we can run it:
+
+```
+python3 test.py
+```
+
+### Trigger the tests via docker healthcheck
+
+In the `docker-compose.yml` file, uncomment the healthcheck section with configuration changes if you prefer the tests to be triggered automatically against the running container on a periodic basis.
+
+```
+# Uncommnet the healthcheck section with desired configuration settings to determine if the state of the container is healthy
+# This healthcheck triggers the python tests based on the options: interval, timeout and start_period
+# healthcheck:
+#   test: ["CMD", "python3", "/usr/src/app/integration-test/test.py"]
+#   interval: 5m30s
+#   timeout: 30s
+#   retries: 3
+#   start_period: 30s
+```

docker-compose.yml

@@ -5,7 +5,7 @@ services:
     build:
       context: ./dphe-stream
     # Build the image with name and tag
-    image: dphe-stream:0.1.0-cr
+    image: dphe-stream:0.2.0-cr
     hostname: dphe-stream
     container_name: dphe-stream
     init: true
@@ -19,14 +19,14 @@ services:
       - "8181:8080"
     volumes:
       # Make log accessible on host
-      - "./dphe-stream/logs:/usr/src/app/v0.1.0-cr-release/dphe-stream-rest/logs"
+      - "./dphe-stream/logs:/usr/src/app/v0.2.0-cr-release/dphe-stream-rest/logs"
     networks:
       - dphe-stream-network
 
   dphe-stream-nginx:
     build: ./dphe-stream-nginx
     # Build the image with name and tag
-    image: dphe-stream-nginx:0.1.0-cr
+    image: dphe-stream-nginx:0.2.0-cr
     hostname: dphe-stream-nginx
     container_name: dphe-stream-nginx
     init: true
@@ -37,6 +37,14 @@ services:
     ports:
       # Map host machine port 8080 to container port 8080 (since nginx runs as non-root)
       - "8080:8080"
+    # Uncommnet the healthcheck section with desired configuration settings to determine if the state of the container is healthy
+    # This healthcheck triggers the python tests based on the options: interval, timeout and start_period
+    # healthcheck:
+    #   test: ["CMD", "python3", "/usr/src/app/integration-test/test.py"]
+    #   interval: 5m30s
+    #   timeout: 30s
+    #   retries: 3
+    #   start_period: 30s
     volumes:
       # Make log accessible on host
       - "./dphe-stream-nginx/logs:/usr/src/app/logs"

dphe-stream-nginx/integration-test/reports/fake_patient1/fake_patient1_doc10_NOTE.txt

@@ -0,0 +1,21 @@
+===================================================================
+Report ID.....................10,doc10
+Patient ID....................pt123123123
+Patient Name..................Fake Patient1
+Principal Date................20110301 1015
+Record Type...................NOTE
+Patient DOB...................04/01/1960
+
+
+CLINICAL HISTORY: The patient is a 51 year old woman with newly diagnosed invasive lobular carcinoma of the left breast. She is ER positive, PR positive and HER2/NEU negative. 
+
+Briefly the patient was diagnosed in January of 2010 with metastatic breast cancer of the right breast. She was stage 2A (T1, N1, M0). She received chemotherapy with Adriamycin and Carboplatin followed by Abraxane. She underwent a right mastectomy with axillary lymph node dissection on 7/10/10. The surgical margins were negative and she received radiation therapy to the right breast and axilla area. She is being maintained on oral Tamoxifen.
+
+INTERIM HISTORY: She noticed a small nodule in her left breast. Mammography obtained on 1/31/10 revealed a firm mass in the lower outer left breast position that measures 1x2 cm. Ultrasound performed on 1/31/10 showed a 1.1x1.2x2.1 cm hypoechoic mass at the 4 o’clock region. The mass was highly suggestive of a malignancy. She underwent a left breast segmental mastectomy with left axillary lymph node biopsy on 2/1/11. Pathology came back as stage IA (pT1a, pN0) invasive lobular carcinoma with a focus of lobular carcinoma in situ of the left breast, lower outer quadrant, 4 o’clock position.
+
+
+Physical Exam: The patient’s physical examination shows a well healed left surgical incision. The previous incision of the right breast is without masses. There are no signs of infection, swelling or erythema. She denies any pain, fever, nausea or vomiting.
+
+Impression: A 51 year old woman with stage 1A (pT1a, pN0) invasive lobular carcinoma and LCIS of the left breast S/P segmental mastectomy and left axillary lymph node biopsy.  
+
+Recommendations: She will undergo radiation therapy to the left breast.

dphe-stream-nginx/integration-test/reports/fake_patient1/fake_patient1_doc1_RAD.txt

@@ -0,0 +1,22 @@
+===================================================================
+Report ID.....................1,doc1
+Patient ID....................pt123123123
+Patient Name..................Fake Patient1
+Principal Date................20100123 1045
+Record Type...................RAD
+Patient DOB...................04/01/1960
+
+CLINICAL HISTORY:
+This is a 50 year old peri-menopausal female who underwent mammogram on 1/28/09 for a palpable lump in the right breast. Ultrasonography revealed a 1.2x3.4x5.6 cm hypoechoic mass in the upper inner quadrant at the 1 0’clock position. Ultrasonography also revealed an abnormally thickened lymph node in the right axilla which had a thickened cortex of 7 mm.  She now presents for U/S guided core biopsy of the mass and the abnormal lymph node.
+PROCEDURE: 
+Ultrasound guided core biopsy of right breast 1 o’clock abnormality with clip placement
+Ultrasound guided core biopsy of right axilla abnormal lymph node with clip placement
+
+FINDINGS:
+The right breast and axilla were sterilely prepped and draped in the usual standard fashion.  First the right 1 o’clock position 5 cm from the nipple was targeted.  Local anesthesia was obtained with 2% xylocaine.  A small skin incision was made.  Under ultrasound guidance from a medial approach, 2 passes with a 14 gauge biopsy device were performed and sent to pathology.  A clip was placed. 
+Then attention was turned to the right abnormal axillary lymph node.  Local anesthesia was obtained with 2% xylocaine.  A small skin incision was made.  Under ultrasound guidance from an inferomedial approach, 2 passes with a 14 gauge biopsy device were performed and sent to pathology.  A  clip was placed at the site of the biopsy.
+The wounds were cleaned and dressed.  The patient tolerated the procedure well, and there were no complications.
+Post procedure mammogram of the right breast demonstrated adequate clip placement.
+IMPRESSION:
+Uncomplicated ultrasound guided core biopsies of the right breast at the 1 o’clock position and abnormal right axillary lymph node.
+

dphe-stream-nginx/integration-test/reports/fake_patient1/fake_patient1_doc2_SP.txt

@@ -0,0 +1,38 @@
+===================================================================
+Report ID.....................2,doc2
+Patient ID....................pt123123123
+Patient Name..................Fake Patient1
+Principal Date................20100123 1315
+Record Type...................SP
+Patient DOB...................04/01/1960
+
+
+
+CLINICAL HISTORY:
+This 50 year old peri-menopausal female is S/P ultrasound guided core biopsy of a suspicious lump in the right breast and an abnormal axillary lymph node.  
+
+
+
+
+FINAL DIAGNOSIS:
+PART 1:  RIGHT BREAST ULTRASOUND GUIDED CORE BIOPSY
+A.	INFILTRATING DUCTAL CARCINOMA, NUCLEAR GRADE 3
+B.    FOCAL DUCTAL CARCINOMA IN SITU, NUCLEAR GRADE 2
+
+PART 2: RIGHT AXILLARY LYMPH NODE ULTRASOUND GUIDED CORE BIOPSY
+A. METASTATIC CARCINOMA INVLOVING LYMPH NODE CORES
+
+
+IMMUNOHISTOCHEMISTRY:
+
+RESULT						H-SCORE
+
+ESTROGEN RECEPTOR - NEGATIVE		0
+
+PROGESTERONE RECEPTOR - NEGATIVE		0
+
+HER2/NEU - NEGATIVE				0
+
+
+
+

dphe-stream-nginx/integration-test/reports/fake_patient1/fake_patient1_doc3_NOTE.txt

@@ -0,0 +1,24 @@
+===================================================================
+Report ID.....................3,doc3
+Patient ID....................pt123123123
+Patient Name..................Fake Patient1
+Principal Date................20100131 1015
+Record Type...................NOTE
+Patient DOB...................04/01/1960
+
+CLINICAL HISTORY:
+This is a 50 year old peri-menopausal female with clinical stage 2A (T1,N2,M0) triple negative infiltrating ductal carcinoma and DCIS of the right breast. An enlarged right axillary lymph node was biopsied and found to be positive for metastatic disease. MRI revealed a tumor 1.9 cm in diameter. She presents now to discuss options for treatment.
+
+INTERIM HISTORY:
+Patient seen post biopsy for evaluation and management of newly diagnosed infiltrating ductal carcinoma of the right breast with a focus of DCIS. She underwent a U/S core biopsy of the right breast and lymph node on 1/23/10 which revealed infiltrating ductal carcinoma. Breast MRI on 1/25/10 demonstrated a right breast malignancy at the 1 o’clock position measuring 1.2x3.4x5.6 cm. MRI also showed a left axillary lymphadenopathy. MRI of the left breast was without malignancy. PET/CT on 1/26/10 showed no evidence of distant metastases.
+Physical Exam: Well appearing female in NAD
+VS: BP 120/80, Pulse 71, Temp 97.0
+HEENT: NC/AT, PERRL, EOMI, sclera non-icteric
+Neck: Supple, no masses, thyroid not enlarged
+LUNGS: CTA, No wheezes, rales, rhonchi
+COR: RRR, No murmurs, rubs or gallops
+ABD: Soft, non-tender, non-distended, No HSM.
+EXT: Good pulses, No clubbing, cyanosis or edema
+
+IMPRESSION: 
+On review of the patient’s screening and diagnostic imaging and biopsy results she has a stage IIA (T1,N2,M0) infiltrating ductal carcinoma of the right breast with metastatic lymph node in the right axilla. She will be treated with neoadjuvant chemotherapy consisting of Adriamycin and Carboplatin for 4 cycles followed by weekly Abraxane. We will obtain a mammography midway through therapy to assess tumor response. 

dphe-stream-nginx/integration-test/reports/fake_patient1/fake_patient1_doc4_NOTE.txt

@@ -0,0 +1,21 @@
+===================================================================
+Report ID.....................4,doc4
+Patient ID....................pt123123123
+Patient Name..................Fake Patient1
+Principal Date................20100531 1145
+Record Type...................NOTE
+Patient DOB...................04/01/1960
+
+DIAGNOSIS: Stage 2A (T1,N2,M0) infiltrating ductal carcinoma of the right breast triple negative. 
+INTERIM HISTORY: Patient returns for ongoing management of biopsy proven right breast cancer with axillary lymph node metastasis. Her initial visit was on 1/21/2010. Breast examination at that time was significant for a 2 cm palpable lump in the upper inner quadrant of the right breast at the 1 o’clock position. She also had multiple abnormally enlarged right axillary lymph nodes. She has completed 4 cycles of Adriamycin and Carboplatin. She is currently undergoing therapy with Abraxane and has completed 9 of 12 cycles. She has tolerated her chemotherapy well without nausea, vomiting or diarrhea. Breast MRI on 5/25/10 revealed the mass at the 1 o’clock position measuring 1.0x2.9x3.5 cm from previous 1.2x3.4x5.6 cm. The metastatic right axillary lymph node now measures 4.0 mm from previous 7.0 mm. PET/CT on 5/26/10 showed no evidence of distant metastases.
+Physical Exam: Well appearing female in NAD
+VS: BP 120/80, Pulse 72, Temp 97.2
+HEENT: NC/AT, PERRL, EOMI, sclera non-icteric
+Neck: Supple, no masses, thyroid not enlarged
+LUNGS: CTA, No wheezes, rales, rhonchi
+COR: RRR, No murmurs, rubs or gallops
+ABD: Soft, non-tender, non-distended, No HSM.
+EXT: Good pulses, No clubbing, cyanosis or edema
+
+IMPRESSION: 
+Patient has had a partial response to neoadjuvant chemotherapy for triple negative infiltrating ductal carcinoma of the right breast.