Hg19 and hg38 analysis from same workflow branch

README.md

@@ -1,6 +1,6 @@
 # DKFZ SNVCalling Workflow
 
-An SNV calling workflow developed in the Applied Bioinformatics and Theoretical Bioinformatics groups at the DKFZ. An earlier version (pre Github) of this workflow was used in the [Pancancer](https://dockstore.org/containers/quay.io/pancancer/pcawg-dkfz-workflow) project. The workflow is only suited for human data (hg37/hg19; some later versions also hg38), because of the big role annotations play in this workflow.
+An SNV calling workflow developed in the Applied Bioinformatics and Theoretical Bioinformatics groups at the DKFZ. An earlier version (pre Github) of this workflow was used in the [Pancancer](https://dockstore.org/containers/quay.io/pancancer/pcawg-dkfz-workflow) project. The workflow is only suited for human data (hg37/GRCh37/hg19; and also for hg38/GRCh38), because of the big role annotations play in this workflow.
 
 > <table><tr><td><a href="https://www.denbi.de/"><img src="docs/images/denbi.png" alt="de.NBI logo" width="300" align="left"></a></td><td><strong>Your opinion matters!</strong> The development of this workflow is supported by the <a href="https://www.denbi.de/">German Network for Bioinformatic Infrastructure (de.NBI)</a>. By completing <a href="https://www.surveymonkey.de/r/denbi-service?sc=hd-hub&tool=SNVCallingWorkflow">this very short (30-60 seconds) survey</a> you support our efforts to improve this tool.</td></tr></table>
 
@@ -114,6 +114,8 @@ Please have a look at the `resources/configurationFiles/analysisSNVCalling.xml`
 
 ## Example Call
 
+For hg19,
+
 ```bash
 roddy.sh run projectConfigurationName@analysisName patientId \
   --useconfig=/path/to/your/applicationProperties.ini \
@@ -123,6 +125,22 @@ roddy.sh run projectConfigurationName@analysisName patientId \
   --cvalues="bamfile_list:/path/to/your/control.bam;/path/to/your/tumor.bam,sample_list:normal;tumor,possibleTumorSampleNamePrefixes:tumor,possibleControlSampleNamePrefixes:normal,REFERENCE_GENOME:/reference/data/hs37d5_PhiX.fa,CHROMOSOME_LENGTH_FILE:/reference/data/hs37d5_PhiX.chromSizes,extractSamplesFromOutputFiles:false"
 ```
 
+For hg38, add the following workflow configs to the `projectConfigs`
+
+```xml
+  <availableAnalyses>
+    <analysis id="snvCalling" configuration="snvCallingAnalysisGRCh38" useplugin="SNVCallingWorkflow:3.0.0"/>
+  </availableAnalyses>
+```
+
+```bash
+roddy.sh run projectConfigurationName@analysisName patientId \
+  --useconfig=/path/to/your/applicationProperties.ini \
+  --configurationDirectories=/path/to/your/projectConfigs \
+  --useiodir=/input/directory,/output/directory/snv \
+  --cvalues="bamfile_list:/path/to/your/control.bam;/path/to/your/tumor.bam,sample_list:normal;tumor,possibleTumorSampleNamePrefixes:tumor,possibleControlSampleNamePrefixes:normal,REFERENCE_GENOME:/reference/data/GRCh38_decoy_ALT_HLA_PhiX.fa,CHROMOSOME_LENGTH_FILE:/reference/data/GRCh38_decoy_ALT_HLA_PhiX.chromSizes,extractSamplesFromOutputFiles:false"
+```
+
 ### No Control
 
 * Set the parameter `isNoControlWorkflow` to `true`.
@@ -157,6 +175,20 @@ The optional configuration JSON file defaults to the `convertToStdVCF.json` resi
 
   * patch: Remove all code related to PyPy and hts-python (including `copysam.py` and `PYPY_OR_PYTHON_BINARY`)
 
+
+* 3.0.0
+
+  * Major
+    * Support for hg38/GRCh38 reference genome and variant calling from ALT and HLA contigs.
+  * Minor
+    * For hg38: Removing mappability and repeat elements' annotations from penalty calculations.
+    * skipREMAP: Option to remove repeat elements and mappability from confidence annotations in hg19.
+    * Removing EVS And ExAC AF from the annotations and no-control workflow filtering
+    * Support for variant calling from CRAM files
+    * Bug fix: Removing "quote" around the raw filter option `<RAW_SNV_FILTER_OPTIONS>`
+    * Update `COWorkflowsBasePlugin` to `1.4.2`
+
+
 * 2.2.0
 
   * minor: Update virtualenv

buildinfo.txt

@@ -1,4 +1,4 @@
-dependson=COWorkflowsBasePlugin:1.3.0
+dependson=COWorkflowsBasePlugin:1.4.2
 JDKVersion=1.8
 GroovyVersion=2.4
 RoddyAPIVersion=3.4

buildversion.txt

@@ -1,2 +1,2 @@
-2.1
+3.0
 0

resources/analysisTools/snvPipeline/PurityReloaded.py

@@ -251,6 +251,10 @@ def parseVcf(file,num):
 	while (l!= ""):
 		t=l.split('\t')
 		if (t[0][0] != "#") and isValid(t):
+			# Skipping the non-primary assembly variants from purity calculations
+			if t[0].startswith('HLA') or t[0].endswith('_alt'):
+				l=vcf.readline()
+				continue
 			i = chromMap[t[0]]
 			if (t[12]=="germline"):
 			  #DP5=string.split(string.split(t[11],";")[1],",")

resources/analysisTools/snvPipeline/createErrorPlots.py

@@ -31,7 +31,7 @@ def calculateLogSize(size, max_val, base):
 	return math.log(((size/max_val)*(base-1.))+1., base)
 
 def calculateRootedSize(size, max_val):
-	if(float(size) != 0.0):
+	if(float(size) != 0.0 and float(max_val) != 0.0):
 		return np.sqrt(size/max_val)
 	else:
 		return 0.0
@@ -199,6 +199,10 @@ def calculateErrorMatrix(vcfFilename, referenceFilename, errorType):
 			# 23.05.2016 JB: Excluded multiallelic SNVs
 			if ',' in split_line[header.index("ALT")]: continue
 
+			# 21.02.2023 NP: Excluded SNVs with 'N' before or after "," in context
+			if {'N,', ',N'}.intersection(split_line[header.index("SEQUENCE_CONTEXT")]):
+				continue
+
 			chrom = split_line[header.index("CHROM")]
 			pos = int(split_line[header.index("POS")])
 			context = ""