-
Notifications
You must be signed in to change notification settings - Fork 18
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into phillips/epic_enum
- Loading branch information
Showing
4 changed files
with
149 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,7 +3,7 @@ title: GeoMx (NGS) | |
| schema_name: geomx-ngs | ||
| category: Spatial Transcriptomics | ||
| all_versions_deprecated: False | ||
| exclude_from_index: True | ||
| exclude_from_index: False | ||
| layout: default | ||
|
|
||
| --- | ||
|
|
@@ -28,7 +28,38 @@ Related files: | |
| <br> | ||
|
|
||
| ## Directory schemas | ||
| <summary><b>Version 2.0 (use this one)</b></summary> | ||
| <summary><b>Version 2.1 (use this one)</b></summary> | ||
|
|
||
| | pattern | required? | description | | ||
| | --- | --- | --- | | ||
| | <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. | | ||
| | <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. | | ||
| | <code>extras\/microscope_settings\.json</code> | | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. | | ||
| | <code>raw\/.*</code> | ✓ | All raw data files for the experiment. | | ||
| | <code>raw\/[^\/]+_LabWorksheet.txt</code> | ✓ | An Excel spreadsheet to refer to in setting up the library. This file documents all of the samples from a single collection plate. Generated by DSP run, prior to sequencing. | | ||
| | <code>raw\/[^\/]+_config\.ini</code> | ✓ | Needed to generate the DCC file from the fastq file. Contains pipeline processing parameters. Generated by DSP run, prior to sequencing. | | ||
| | <code>raw\/[^\/]+_SeqCodeIndices\.csv</code> | ✓ | A file with sample information needed by the Illumina software. Use the contents of the SeqCodeIndices.csv file to create a SampleSheet.csv for input to the Illumina sequencer. (NextSeq 1000/2000 users download a SampleSheet.csv and whitelist.txt instead of SeqCodeIndices.csv.) Generated by DSP run. | | ||
| | <code>raw\/markers\.csv</code> | | A csv file describing any morphology markers used to guide ROI and/or AOI selection [this should be similar in structure to the antibodies file] | | ||
| | <code>raw\/[^\/]*targets\.pkc</code> | ✓ | The file listing probe barcode sequence and corresponding gene symbol or proteins targeted by that probe. This should be consistent for the same probe panel. | | ||
| | <code>raw\/additional_panels_used\.csv</code> | | If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include:manufacturer, model/name, product code. | | ||
| | <code>raw\/custom_probe_set\.csv</code> | ✓ | This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include:target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>). | | ||
| | <code>raw\/fastq\/.*</code> | ✓ | Raw sequencing files for the experiment | | ||
| | <code>raw\/fastq\/oligo\/.*</code> | ✓ | Directory containing fastq files pertaining to oligo sequencing. | | ||
| | <code>raw\/fastq\/oligo\/[^\/]+\.fastq\.gz</code> | ✓ | This is a gzip version of the fastq file. This file contains the cell barcode and unique molecular identifier (technical). | | ||
| | <code>raw\/images\/.*</code> | | Directory containing raw image files. This directory should include at least one raw file. | | ||
| | <code>raw\/images\/overlay\.(?:jpeg|tiff)</code> | | State whether an overlay image was used to guide ROI selection. If an overlay is used, then the overlay details will be provided in the protocols.io protocol. If used, this needs to be uploaded. It is not included in the OME TIFF. This can be a JPEG or TIFF file | | ||
| | <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. | | ||
| | <code>lab_processed\/Initial\s{1}Dataset\.xlsx</code> | ✓ | **[QA/QC]** An excel spreadsheet that is downloaded from the GeoMx DSP Data Analysis Suite containing QA/QC metrics based on raw, unprocessed target counts. This file contains one row per AOI/segment and no analyses span AOI. The AOIs included in this file can come from different GeoMx runs and hence span Globus uploads. So care must be taken to make sure the appropriate AOIs are included in the file. | | ||
| | <code>lab_processed\/annotations\.xlsx</code> | | AOI specific annotations. This might include cell type and anatomical information. | | ||
| | <code>lab_processed\/dcc\/.*</code> | ✓ | DCC files generated from fastq by the Nanostring GeoMx NGS Pipeline. | | ||
| | <code>lab_processed\/dcc\/[^\/]+\.dcc</code> | ✓ | DCC files containing target probe counts, generated from fastq by the Nanostring GeoMx NGS Pipeline. | | ||
| | <code>lab_processed\/images\/.*</code> | ✓ | Processed image files | | ||
| | <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> | | ||
| | <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> | | ||
| | <code>lab_processed\/primary_analysis\/.*</code> | ✓ | Primary analysis results | | ||
| | <code>lab_processed\/primary_analysis\/Q3\s{1}Normalized\.xlsx</code> | ✓ | **[QA/QC]** Results from initial procesing by GeoMx DSP Data Analysis Suite. The collection of datasets were normalized using Q3 normalization after target genes below the limit of quantitation (LOQ) are removed. | | ||
|
|
||
| <summary><b>Version 2.0</b></summary> | ||
|
|
||
| | pattern | required? | description | | ||
| | --- | --- | --- | | ||
|
|
||
115 changes: 115 additions & 0 deletions
115
src/ingest_validation_tools/directory-schemas/geomx-ngs-v2.1.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| files: | ||
| - | ||
| pattern: extras\/.* | ||
| required: True | ||
| description: Folder for general lab-specific files related to the dataset. | ||
| - | ||
| pattern: extras\/microscope_hardware\.json | ||
| required: True | ||
| description: A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. | ||
| is_qa_qc: True | ||
| - | ||
| pattern: extras\/microscope_settings\.json | ||
| required: False | ||
| description: A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. | ||
| is_qa_qc: True | ||
| - | ||
| pattern: raw\/.* | ||
| required: True | ||
| description: All raw data files for the experiment. | ||
| - | ||
| pattern: raw\/[^\/]+_LabWorksheet.txt | ||
| required: True | ||
| description: An Excel spreadsheet to refer to in setting up the library. This file documents all of the samples from a single collection plate. Generated by DSP run, prior to sequencing. | ||
| - | ||
| pattern: raw\/[^\/]+_config\.ini | ||
| required: True | ||
| description: Needed to generate the DCC file from the fastq file. Contains pipeline processing parameters. Generated by DSP run, prior to sequencing. | ||
| - | ||
| pattern: raw\/[^\/]+_SeqCodeIndices\.csv | ||
| required: True | ||
| description: A file with sample information needed by the Illumina software. Use the contents of the SeqCodeIndices.csv file to create a SampleSheet.csv for input to the Illumina sequencer. (NextSeq 1000/2000 users download a SampleSheet.csv and whitelist.txt instead of SeqCodeIndices.csv.) Generated by DSP run. | ||
| - | ||
| pattern: raw\/markers\.csv | ||
| required: False | ||
| description: A csv file describing any morphology markers used to guide ROI and/or AOI selection [this should be similar in structure to the antibodies file] | ||
| - | ||
| pattern: raw\/[^\/]*targets\.pkc | ||
| required: True | ||
| description: The file listing probe barcode sequence and corresponding gene symbol or proteins targeted by that probe. This should be consistent for the same probe panel. | ||
| - | ||
| pattern: raw\/additional_panels_used\.csv | ||
| required: False | ||
| description: If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include:manufacturer, model/name, product code. | ||
| - | ||
| pattern: raw\/custom_probe_set\.csv | ||
| required: True | ||
| description: This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include:target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>). | ||
| - | ||
| pattern: raw\/fastq\/.* | ||
| required: True | ||
| description: Raw sequencing files for the experiment | ||
| - | ||
| pattern: raw\/fastq\/oligo\/.* | ||
| required: True | ||
| description: Directory containing fastq files pertaining to oligo sequencing. | ||
| - | ||
| pattern: raw\/fastq\/oligo\/[^\/]+\.fastq\.gz | ||
| required: True | ||
| description: This is a gzip version of the fastq file. This file contains the cell barcode and unique molecular identifier (technical). | ||
| is_qa_qc: False | ||
| - | ||
| pattern: raw\/images\/.* | ||
| required: False | ||
| description: Directory containing raw image files. This directory should include at least one raw file. | ||
| - | ||
| pattern: raw\/images\/overlay\.(?:jpeg|tiff) | ||
| required: False | ||
| description: State whether an overlay image was used to guide ROI selection. If an overlay is used, then the overlay details will be provided in the protocols.io protocol. If used, this needs to be uploaded. It is not included in the OME TIFF. This can be a JPEG or TIFF file | ||
| - | ||
| pattern: lab_processed\/.* | ||
| required: True | ||
| description: Experiment files that were processed by the lab generating the data. | ||
| - | ||
| pattern: lab_processed\/Initial\s{1}Dataset\.xlsx | ||
| required: True | ||
| description: An excel spreadsheet that is downloaded from the GeoMx DSP Data Analysis Suite containing QA/QC metrics based on raw, unprocessed target counts. This file contains one row per AOI/segment and no analyses span AOI. The AOIs included in this file can come from different GeoMx runs and hence span Globus uploads. So care must be taken to make sure the appropriate AOIs are included in the file. | ||
| is_qa_qc: True | ||
| - | ||
| pattern: lab_processed\/annotations\.xlsx | ||
| required: False | ||
| description: AOI specific annotations. This might include cell type and anatomical information. | ||
| - | ||
| pattern: lab_processed\/dcc\/.* | ||
| required: True | ||
| description: DCC files generated from fastq by the Nanostring GeoMx NGS Pipeline. | ||
| - | ||
| pattern: lab_processed\/dcc\/[^\/]+\.dcc | ||
| required: True | ||
| description: DCC files containing target probe counts, generated from fastq by the Nanostring GeoMx NGS Pipeline. | ||
| - | ||
| pattern: lab_processed\/images\/.* | ||
| required: True | ||
| description: Processed image files | ||
| - | ||
| pattern: lab_processed\/images\/[^\/]+\.ome\.tiff | ||
| required: True | ||
| description: OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> | ||
| is_qa_qc: False | ||
| example: lab_processed/images/HBM892.MDXS.293.ome.tiff | ||
| - | ||
| pattern: lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv | ||
| required: True | ||
| description: This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> | ||
| is_qa_qc: False | ||
| - | ||
| pattern: lab_processed\/primary_analysis\/.* | ||
| required: True | ||
| description: Primary analysis results | ||
| - | ||
| pattern: lab_processed\/primary_analysis\/Q3\s{1}Normalized\.xlsx | ||
| required: True | ||
| description: Results from initial procesing by GeoMx DSP Data Analysis Suite. The collection of datasets were normalized using Q3 normalization after target genes below the limit of quantitation (LOQ) are removed. | ||
| is_qa_qc: True | ||
|
|
||
|
|
1 change: 0 additions & 1 deletion
1
src/ingest_validation_tools/table-schemas/assays/geomx-ngs-v2.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters