Skip to content

Commit

Permalink
merging from main
Browse files Browse the repository at this point in the history
  • Loading branch information
@gesinaphillips
gesinaphillips committed Aug 28, 2024
2 parents 67b1f74 + 31f3cf3 commit af88221
Show file tree
Hide file tree
Showing 17 changed files with 631 additions and 44 deletions.
13 changes: 12 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,17 @@
# Changelog

## v0.0.24
## v0.0.24 (in progress)
- Release MERFISH
- Add MERFISH directory schema
- Fix documentation issue for MERFISH
- Add CEDAR link for MERFISH
- Update MERFISH directory schema
- Update Phenocycler docs
- Update MERFISH directory schema
- Add next-gen Cell DIVE directory schema
- Update MIBI directory schema
- Update Visium no probes directory schema
- Add Cell DIVE to index
- Change to EntityTypeInfo constraint format to support constraints endpoint

## v0.0.23
Expand Down
23 changes: 21 additions & 2 deletions docs/celldive/current/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Cell DIVE
schema_name: celldive
category: Multiplex Fluorescence Based Experiment (MxFBE)
all_versions_deprecated: False
exclude_from_index: True
exclude_from_index: False
layout: default

---
Expand All @@ -28,5 +28,24 @@ Related files:
<br>

## Directory schemas
<summary><a href="https://docs.google.com/spreadsheets/d/1pZD2e51e4QkxzIk6xjHPPu1RBZpx5mzoykMmlaDK8rA"><b>Version 2.0 (use this one)</b> (draft - submission of data prepared using this schema will be supported by Sept. 30) </a></summary>
<summary><b>Version 2.0 (use this one)</b></summary>

| pattern | required? | description |
| --- | --- | --- |
| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. [Exists in all assays] |
| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
| <code>extras\/microscope_settings\.json</code> | | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
| <code>raw\/.*</code> | ✓ | This is a directory containing raw data. |
| <code>raw\/images\/.*</code> | ✓ | Raw image files. Using this subdirectory allows for harmonization with other more complex assays, like Visium that includes both raw imaging and sequencing data. |
| <code>raw\/images\/round_info_[^\/]+\.dat</code> (example: <code>raw/images/round_info_002.dat</code>) | ✓ | Metadata file for the capture item-value tab separated format. This contains various instrument and acquisition details for each acquisition cycle. |
| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
| <code>lab_processed\/images\/.*</code> | ✓ | This is a directory containing processed image files |
| <code>lab_processed\/images\/region_[^\/]+\/[^\/]+_region_[^\/]+\.ome\.(?:tif&#124;tiff)</code> (example: <code>lab_processed/images/region_001/S20030092_region_011.ome.tif</code>) | ✓ | OME TIFF Files for the corresponding region (e.g. region_001) by slide (e.g S20030077), organized into subdirectories based on their region. |
| <code>lab_processed\/images\/region_[^\/]+\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> |
| <code>lab_processed\/annotations\/.*</code> | ✓ | This is a directory containing annotations. |
| <code>lab_processed\/annotations\/slide_list\.txt</code> | ✓ | Information about the slides used by the experiment- each line corresponds to a slide name (begins with S - e.g. S20030077) - used in filenames. |
| <code>lab_processed\/virtual_histology\/.*</code> | ✓ | This is a directory containing annotations for virtual histology images |
| <code>lab_processed\/virtual_histology\/HandE_RGB_thumbnail\.jpg</code> | ✓ | Virtual H&E RGB thumbnail |
| <code>lab_processed\/virtual_histology\/HandE_RGB\.tif</code> | ✓ | Virtual H&E RGB image |
| <code>lab_processed\/virtual_histology\/[^\/]+_VHE_region_[^\/]+\.tif</code> | ✓ | Virtual H&E image |

86 changes: 80 additions & 6 deletions docs/merfish/current/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,105 @@ title: MERFISH
schema_name: merfish
category: Fluorescence In Situ Hybridization (FISH)
all_versions_deprecated: False
exclude_from_index: True
exclude_from_index: False
layout: default

---
Prepare your metadata based on the latest metadata schema using one of the template files below. See the instructions in the [Metadata Validation Workflow](https://docs.google.com/document/d/1lfgiDGbyO4K4Hz1FMsJjmJd9RdwjShtJqFYNwKpbcZY) document for more information on preparing and validating your metadata.tsv file prior to submission.

Related files:

Excel and TSV templates for this schema will be available when the draft next-generation schema, to be used in all future submissions, is finalized (no later than Sept. 30).

- [📝 Excel template](https://raw.githubusercontent.com/hubmapconsortium/dataset-metadata-spreadsheet/main/merfish/latest/merfish.xlsx): For metadata entry.
- [📝 TSV template](https://raw.githubusercontent.com/hubmapconsortium/dataset-metadata-spreadsheet/main/merfish/latest/merfish.tsv): Alternative for metadata entry.


[This link](https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0) lists the set of fields that are required in the OME TIFF file XML header.

## Metadata schema


<summary><b>Version 2 (use this one)</b> (draft - submission of data prepared using this schema will be supported by Sept. 30) (TBD)</summary>
<summary><a href="https://openview.metadatacenter.org/templates/https:%2F%2Frepo.metadatacenter.org%2Ftemplates%2Ff1ef260f-d4a3-43db-a739-49b394aeee20"><b>Version 2 (use this one)</b></a></summary>



<br>

## Directory schemas
<summary><b>Version 2.0 (use this one)</b></summary>
<summary><b>Version 2.2 (use this one)</b></summary>

| pattern | required? | description |
| --- | --- | --- |
| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. |
| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
| <code>extras\/microscope_settings\.json</code> | | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
| <code>raw\/.*</code> | ✓ | All raw data files for the experiment. |
| <code>raw\/additional_panels_used\.csv</code> | | If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include:manufacturer, model/name, product code. |
| <code>raw\/gene_panel\.csv</code> | ✓ | The list of target genes. The expected format is gene_id (ensembl ID), gene_name. |
| <code>raw\/custom_probe_set\.csv</code> | | This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include:target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>). |
| <code>raw\/micron_to_mosaic_pixel_transform\.csv</code> | | Matrix used to transform from pixels to physical distance. |
| <code>raw\/manifest\.json</code> | ✓ | This file contains stain by channel details and pixel details. |
| <code>raw\/codebook\.csv</code> | ✓ | CSV containing codebook information for the experiment. Rows are barcodes and columns are imaging rounds. The first column is the barcode target, and the following column IDs are expected to be sequential, and round identifiers are expected to be integers (not roman numerals). |
| <code>raw\/positions\.csv</code> | ✓ | File that includes the top left coordinate of each tiled image. This is required to stitch the images. |
| <code>raw\/dataorganization\.csv</code> | ✓ | Necessary image definitions |
| <code>raw\/[^\/]+\.DAX</code> | ✓ | The raw image stack. |
| <code>raw\/images\/.*</code> | ✓ | Directory containing raw image files. This directory should include at least one raw file. |
| <code>raw\/images\/[^\/]+\.tif</code> | ✓ | Raw microscope file for the experiment. |
| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
| <code>lab_processed\/detected_transcripts\.csv</code> | ✓ | A file containing the locations of each RNA target. |
| <code>lab_processed\/images\/.*</code> | ✓ | Processed image files |
| <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> |
| <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> |

<summary><b>Version 2.1</b></summary>

| pattern | required? | description |
| --- | --- | --- |
| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. |
| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
| <code>extras\/microscope_settings\.json</code> | | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
| <code>raw\/.*</code> | ✓ | All raw data files for the experiment. |
| <code>raw\/additional_panels_used\.csv</code> | | If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include:manufacturer, model/name, product code. |
| <code>raw\/gene_panel\.csv</code> | ✓ | The list of target genes. |
| <code>raw\/custom_probe_set\.csv</code> | | This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include:target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>). |
| <code>raw\/micron_to_mosaic_pixel_transform\.csv</code> | | Matrix used to transform from pixels to physical distance. |
| <code>raw\/manifest\.json</code> | ✓ | This file contains stain by channel details and pixel details. |
| <code>raw\/codebook\.csv</code> | ✓ | CSV containing codebook information for the experiment. Rows are barcodes and columns are imaging rounds. The first column is the barcode target, and the following column IDs are expected to be sequential, and round identifiers are expected to be integers (not roman numerals). |
| <code>raw\/positions\.csv</code> | ✓ | File that includes the top left coordinate of each tiled image. This is required to stitch the images. |
| <code>raw\/dataorganization\.csv</code> | ✓ | Necessary image definitions |
| <code>raw\/data\/.*</code> | ✓ | All raw stack data files for the MERFISH experiment. |
| <code>raw\/data\/[^\/]+\.dax</code> | ✓ | The raw image stack. |
| <code>raw\/data\/[^\/]+\.inf</code> | ✓ | Information file with dax image format specifications. Variable expected for downstream processing with PIPEFISH are frame dimensions, number of frames, little/big endian, stage X and Y locations, lock target, scalemin, and scalemax. |
| <code>raw\/images\/.*</code> | ✓ | Directory containing raw image files. This directory should include at least one raw file. |
| <code>raw\/images\/[^\/]+\.tif</code> | ✓ | Raw microscope file for the experiment. |
| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
| <code>lab_processed\/detected_transcripts\.csv</code> | ✓ | A file containing the locations of each RNA target. |
| <code>lab_processed\/images\/.*</code> | ✓ | Processed image files |
| <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> |
| <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> |

<summary><b>Version 2.0</b></summary>

| pattern | required? | description |
| --- | --- | --- |
| <code>TODO</code> | ✓ | Directory structure not yet specified. |
| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. [Exists in all assays] |
| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. |
| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
| <code>extras\/microscope_settings\.json</code> | | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
| <code>raw\/.*</code> | ✓ | All raw data files for the experiment. |
| <code>raw\/additional_panels_used\.csv</code> | | If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include:manufacturer, model/name, product code. |
| <code>raw\/gene_panel\.csv</code> | ✓ | The list of target genes. |
| <code>raw\/custom_probe_set\.csv</code> | | This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include:target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>). |
| <code>raw\/micron_to_mosaic_pixel_transform\.csv</code> | | Matrix used to transform from pixels to physical distance. |
| <code>raw\/manifest\.json</code> | ✓ | This file contains stain by channel details and pixel details. |
| <code>raw\/codebook\.csv</code> | ✓ | CSV containing codebook information for the experiment. Rows are barcodes and columns are imaging rounds. The first column is the barcode target, and the following column IDs are expected to be sequential, and round identifiers are expected to be integers (not roman numerals). |
| <code>raw\/positions\.csv</code> | ✓ | File that includes the top left coordinate of each tiled image. This is required to stitch the images. |
| <code>raw\/dataorganization\.csv</code> | ✓ | Necessary image definitions |
| <code>raw\/[^\/]+\.DAX</code> | ✓ | The raw image stack. |
| <code>raw\/images\/.*</code> | ✓ | Directory containing raw image files. This directory should include at least one raw file. |
| <code>raw\/images\/[^\/]+\.tif</code> | ✓ | Raw microscope file for the experiment. |
| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
| <code>lab_processed\/detected_transcripts\.csv</code> | ✓ | A file containing the locations of each RNA target. |
| <code>lab_processed\/images\/.*</code> | ✓ | Processed image files |
| <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> |
| <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> |

Loading

0 comments on commit af88221

Please sign in to comment.