Adding Support for Metadata #8

emarinier · 2024-04-16T15:05:13Z

Adds support for metadata:

Adds 8 fixed columns that can be renamed with metadata_1_header through metadata_8_header headers.
Adds a module that opens up the TSV-formatted cluster file and appends metadata provided through the samplesheet.
Passes the appended TSV-formatted cluster file to ArborView.

Tested on Azure and it works, but requires manually specifying "av_html":"https://github.com/phac-nml/gasclustering/raw/dev/assets/ArborView.html" as discussed.

github-actions · 2024-04-16T15:06:19Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 9c16037

+| ✅ 139 tests passed       |+
#| ❔  27 tests were ignored |#
!| ❗  15 tests had warnings |!

❗ Test warnings:

schema_lint - Schema $id should be https://raw.githubusercontent.com/phac-nml/gasclustering/master/nextflow_schema.json
Found https://raw.githubusercontent.com/phac-nml/gasclustering/main/nextflow_schema.json
schema_description - No description provided in schema for parameter: pd_outfmt
schema_description - No description provided in schema for parameter: pd_distm
schema_description - No description provided in schema for parameter: pd_missing_threshold
schema_description - No description provided in schema for parameter: pd_sample_quality_threshold
schema_description - No description provided in schema for parameter: pd_match_threshold
schema_description - No description provided in schema for parameter: pd_file_type
schema_description - No description provided in schema for parameter: pd_mapping_file
schema_description - No description provided in schema for parameter: pd_force
schema_description - No description provided in schema for parameter: pd_skip
schema_description - No description provided in schema for parameter: pd_columns
schema_description - No description provided in schema for parameter: pd_count_missing
schema_description - No description provided in schema for parameter: gm_thresholds
schema_description - No description provided in schema for parameter: gm_delimiter
schema_description - No description provided in schema for parameter: av_html

❔ Tests ignored:

files_exist - File is ignored: assets/nf-core-gasclustering_logo_light.png
files_exist - File is ignored: docs/images/nf-core-gasclustering_logo_light.png
files_exist - File is ignored: docs/images/nf-core-gasclustering_logo_dark.png
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: lib/Utils.groovy
files_exist - File is ignored: lib/WorkflowMain.groovy
files_exist - File is ignored: lib/NfcoreTemplate.groovy
files_exist - File is ignored: lib/WorkflowGasclustering.groovy
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-gasclustering_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-gasclustering_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-gasclustering_logo_dark.png
files_unchanged - File ignored due to lint config: docs/README.md
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/gasclustering/gasclustering/.github/workflows/awstest.yml
actions_awsfulltest - actions_awsfulltest
pipeline_name_conventions - pipeline_name_conventions

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-gasclustering_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.0.0dev
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.metadata_1_header= metadata_1
nextflow_config - Config default value correct: params.metadata_2_header= metadata_2
nextflow_config - Config default value correct: params.metadata_3_header= metadata_3
nextflow_config - Config default value correct: params.metadata_4_header= metadata_4
nextflow_config - Config default value correct: params.metadata_5_header= metadata_5
nextflow_config - Config default value correct: params.metadata_6_header= metadata_6
nextflow_config - Config default value correct: params.metadata_7_header= metadata_7
nextflow_config - Config default value correct: params.metadata_8_header= metadata_8
nextflow_config - Config default value correct: params.pd_outfmt= matrix
nextflow_config - Config default value correct: params.pd_distm= scaled
nextflow_config - Config default value correct: params.pd_missing_threshold= 1.0
nextflow_config - Config default value correct: params.pd_sample_quality_threshold= 1.0
nextflow_config - Config default value correct: params.pd_match_threshold= -1.0
nextflow_config - Config default value correct: params.pd_file_type= text
nextflow_config - Config default value correct: params.pd_count_missing= true
nextflow_config - Config default value correct: params.gm_thresholds= 10,5,0
nextflow_config - Config default value correct: params.gm_delimiter= .
nextflow_config - Config default value correct: params.av_html= ./assets/ArborView.html
nextflow_config - Config default value correct: params.max_cpus= 4
nextflow_config - Config default value correct: params.max_memory= 2.GB
nextflow_config - Config default value correct: params.max_time= 1.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
template_strings - Did not find any Jinja template strings (88 files)
schema_lint - Schema lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
multiqc_config - 'assets/multiqc_config.yml' contains export_plots
multiqc_config - 'assets/multiqc_config.yml' contains report_comment
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.13.1
Run at 2024-04-25 20:29:31

assets/schema_input.json

…adata

apetkau

This looks great. Thanks so much @emarinier . I just have a few comments 😄

CHANGELOG.md

assets/schema_input.json

modules/local/appendmetadata/main.nf

tests/pipelines/main.nf.test

…ipeline.

assets/schema_input.json

workflows/gasclustering.nf

apetkau

Thanks so much for all of your changes. I have a few additional comments.

README.md

modules/local/appendmetadata/main.nf

tests/data/samplesheets/samplesheet1.csv

apetkau

I tested in IRIDA Next and this all works

mattheww95

Another wonderful PR! Thank you Eric!!

emarinier added 3 commits April 12, 2024 13:27

First pass solution

d8313fc

Updating basic workflow test.

0b38ae7

Updating changelog.

c214e20

emarinier self-assigned this Apr 16, 2024

emarinier marked this pull request as draft April 16, 2024 15:05

emarinier added 5 commits April 16, 2024 10:17

Updating schema for metadata header parameters.

adc9e4a

Updating README to include metadata information.

940e557

Adding missing test data.

af23d34

prettier

4705b98

prettier

edeb715

emarinier requested review from apetkau and mattheww95 April 18, 2024 15:57

emarinier marked this pull request as ready for review April 18, 2024 15:57

Update gasclustering.nf

4dc4bd2

mattheww95 reviewed Apr 18, 2024

View reviewed changes

assets/schema_input.json Show resolved Hide resolved

emarinier added 2 commits April 22, 2024 10:25

Adding group description to schema.

a46bc2e

Merge branch 'metadata' of github.com:phac-nml/gasclustering into met…

a26fbaf

…adata

apetkau requested changes Apr 24, 2024

View reviewed changes

emarinier added 2 commits April 24, 2024 09:58

Removing old changelog from example pipeline and correcting for new p…

7a563b1

…ipeline.

Convert from example pipeline to gasclustering pipeline README.

1e963e6

apetkau reviewed Apr 24, 2024

View reviewed changes

assets/schema_input.json Show resolved Hide resolved

apetkau reviewed Apr 24, 2024

View reviewed changes

workflows/gasclustering.nf Outdated Show resolved Hide resolved

emarinier added 6 commits April 24, 2024 14:15

Updating schema.

f5c959d

process_low -> process_single

f08030d

More diligence in ensuring rows map to each other correctly.

4155678

Updating ArborView test to be more exact.

1c6fd66

metadata under meta map

a1f69e8

prettier

ed4cadf

apetkau requested changes Apr 25, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

modules/local/appendmetadata/main.nf Show resolved Hide resolved

tests/data/samplesheets/samplesheet1.csv Show resolved Hide resolved

emarinier added 5 commits April 25, 2024 13:01

README grammar

f01999f

Forcing failure when internal tabs exist in metadata columns.

44cfb49

Adding more metadata tests.

d1636fe

Trying to ignore trailing whitespace in test files.

339dc53

Trying to ignore trailing whitespace in test files.

9c16037

apetkau approved these changes Apr 26, 2024

View reviewed changes

apetkau requested a review from mattheww95 April 29, 2024 14:22

mattheww95 approved these changes Apr 29, 2024

View reviewed changes

emarinier merged commit 453246c into dev Apr 29, 2024
4 checks passed

emarinier deleted the metadata branch June 26, 2024 15:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Support for Metadata #8

Adding Support for Metadata #8

emarinier commented Apr 16, 2024 •

edited

Loading

github-actions bot commented Apr 16, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

apetkau left a comment

apetkau left a comment

apetkau left a comment

mattheww95 left a comment

Adding Support for Metadata #8

Adding Support for Metadata #8

Conversation

emarinier commented Apr 16, 2024 • edited Loading

github-actions bot commented Apr 16, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

apetkau left a comment

Choose a reason for hiding this comment

apetkau left a comment

Choose a reason for hiding this comment

apetkau left a comment

Choose a reason for hiding this comment

mattheww95 left a comment

Choose a reason for hiding this comment

emarinier commented Apr 16, 2024 •

edited

Loading

github-actions bot commented Apr 16, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️