Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Param error on latest version of Grandeur #227

Open
DrB-S opened this issue Sep 9, 2024 · 28 comments
Open

Param error on latest version of Grandeur #227

DrB-S opened this issue Sep 9, 2024 · 28 comments

Comments

@DrB-S
Copy link
Contributor

DrB-S commented Sep 9, 2024

I am running the latest version of Grandeur and erroneously getting an error (maxcpus isn't a supported param), even though I am not using that param. Here is my command-line and the error message: ./nextflow run -qs 10 UPHL-BioNGS/Grandeur -profile singularity,msa --iqtree2_outgroup NZ_CP043953.1 --outdir grandeur

N E X T F L O W ~ version 24.04.4

Launching https://github.com/UPHL-BioNGS/Grandeur [lethal_cajal] DSL2 - revision: 35b867a [main]

/^^^^ /^^^^^^^ /^ /^^^ /^^ /^^^^^ /^^^^^^^^ /^^ /^^ /^^^^^^^
/^ /^^ /^^ /^^ /^ ^^ /^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^
/^^ /^^ /^^ /^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^
/^^ /^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^^^^^ /^^ /^^ /^ /^^
/^^ /^^^^ /^^ /^^ /^^^^^^ /^^ /^^ /^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^
/^^ /^ /^^ /^^ /^^ /^^ /^^ /^ ^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^
/^^^^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^^^^ /^^^^^^^^ /^^^^^ /^^ /^^
Currently using the Grandeur workflow for use with microbial sequencing.
The view is great from 8299 feet (2530 meters) above sea level.

Author: Erin Young
email: [email protected]
Version: 3.0.20230205
FATAL: maxcpus isn't a supported param!
Supported params: [outdir, fastas, msa, kraken2_db, mash_db, config_file, reads, sample_sheet, fasta_list, blast_db, blast_db_type, fastani_ref, fastani_ref_list, iqtree2_outgroup, genome_sizes, sra_accessions, minimum_reads, datasets_max_genomes, mash_max_hits, min_core_genes, current_datasets, skip_extras, exclude_top_hit, aligner]

@erinyoung
Copy link
Member

That's... so strange.

There must be a config file or something that has maxcpus set.

I can't find anything in the directory. Is there something in your path by chance?

Also, it looks like you are using version 3.0.20230205 as opposed to the most recent version. Do you know how that could be?

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 10, 2024

Did I improperly pull the latest version of Grandeur with the following? /data/Sequence_analysis/nextflow pull UPHL-BioNGS/Grandeur

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 10, 2024

I received this message after the pull:
Checking UPHL-BioNGS/Grandeur ...
Already-up-to-date - revision: 35b867a [main]

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 10, 2024

Grandeur is on my machine here:
~/.nextflow/assets/UPHL-BioNGS/Grandeur

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 10, 2024

I found it in nextflow.config.

@erinyoung
Copy link
Member

The current version should be 4.5.24250. I wonder why nextflow isn't pulling the latest one.

Perhaps you can delete ~/.nextflow/assets/UPHL-BioNGS/Grandeur and re-pull the workflow?

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 10, 2024 via email

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 10, 2024

I ran the latest version (Version: 4.5.24250), and now I am getting a singularity error:
Pulling Singularity image docker://quay.io/uphl/grandeur_ref:2024-06-26 [cache /data/nextflow_cachedir/quay.io-uphl-grandeur_ref-2024-06-26.img]
ERROR ~ Error executing process > 'average_nucleotide_identity:references'

Caused by:
Failed to pull singularity image
command: singularity pull --name quay.io-uphl-grandeur_ref-2024-06-26.img.pulling.1726010239842 docker://quay.io/uphl/grandeur_ref:2024-06-26 > /dev/null
status : 2
hint : Try and increase singularity.pullTimeout in the config (current is "20m")
message:
/usr/games/singularity:6: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources import load_entry_point
:488: RuntimeWarning: Your system is avx2 capable but pygame was not built with support for it. The performance of some of your blits could be adversely affected. Consider enabling compile time detection with environment variables like PYGAME_DETECT_AVX2=1 if you are compiling without cross compilation.
Usage: singularity [options]

  **singularity: error: no such option: --name**

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

I was able to get a little further in the Grandeur analysis before failure, but it ran into the following 2 errors for Version: 4.5.24250:

  1. genome sizes file missing
    Version: 4.5.24250
    The files and directory for results is grandeur
    [- ] quality_assessment:quast -
    [- ] quality_assessment:mlst -
    [- ] qua…y_assessment:plasmidfinder -
    [- ] min_hash:mash_dist -
    [- ] ave…eotide_identity:references -
    The genome sizes file for this workflow are missing!

  2. prokka.nf error on line 7
    Plus 7 more processes waiting for tasks…
    Pipeline completed at: 2024-09-11T06:46:22.350361222Z
    MultiQC report can be found at grandeur/multiqc/multiqc_report.html
    Summary can be found at grandeur/grandeur_summary.tsv
    Execution status: failed
    ERROR ~ Execution aborted due to an unexpected error

-- Check script '/home/becksts/.nextflow/assets/UPHL-BioNGS/Grandeur/./subworkflows/../modules/local/prokka.nf' at line: 7 or
see '.nextflow.log' file for more details

@erinyoung
Copy link
Member

You encountered these errors at the same time?

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

Yes.

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

I found genome_sizes.json in UPHL-BioNGS/Grandeur/assets/genome_sizes.json
And the size of the reference, Acinetobacter baumannii, is there.

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

Yesterday's errors with singularity were due to nextflow_cachedir being empty on our new server, which I solved by copying the contents of nextflow_cachedir over from our old server.

@erinyoung
Copy link
Member

Is there a way you can share the full error message with me?

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

Sure. I have attached nohup.out. Let me know if you need anything else.
Thanks!
nohup.out.txt
And here is .nextflow.log
nxf.log.txt

@erinyoung
Copy link
Member

It looks like it didn't even run! WTH is this? I wonder if it has anything to do with the iqtree2_outgroup param.


 N E X T F L O W   ~  version 24.04.4

Launching `https://github.com/UPHL-BioNGS/Grandeur` [adoring_boyd] DSL2 - revision: 35b867a8f5 [main]

   /^^^^    /^^^^^^^           /^        /^^^     /^^ /^^^^^     /^^^^^^^^ /^^     /^^ /^^^^^^^    
 /^    /^^  /^^    /^^        /^ ^^      /^ /^^   /^^ /^^   /^^  /^^       /^^     /^^ /^^    /^^  
/^^         /^^    /^^       /^  /^^     /^^ /^^  /^^ /^^    /^^ /^^       /^^     /^^ /^^    /^^  
/^^         /^ /^^          /^^   /^^    /^^  /^^ /^^ /^^    /^^ /^^^^^^   /^^     /^^ /^ /^^      
/^^   /^^^^ /^^  /^^       /^^^^^^ /^^   /^^   /^ /^^ /^^    /^^ /^^       /^^     /^^ /^^  /^^    
 /^^    /^  /^^    /^^    /^^       /^^  /^^    /^ ^^ /^^   /^^  /^^       /^^     /^^ /^^    /^^  
  /^^^^^    /^^      /^^ /^^         /^^ /^^      /^^ /^^^^^     /^^^^^^^^   /^^^^^    /^^      /^^
Currently using the Grandeur workflow for use with microbial sequencing.
The view is great from 8299 feet (2530 meters) above sea level.

Author: Erin Young
email: [email protected]
Version: 4.5.24250
The files and directory for results is grandeur
[-        ] quality_assessment:quast       -
[-        ] quality_assessment:mlst        -
[-        ] qua…y_assessment:plasmidfinder -
[-        ] min_hash:mash_dist             -
[-        ] ave…eotide_identity:references -
The genome sizes file for this workflow are missing!

[-        ] quality_assessment:quast       -
[-        ] quality_assessment:mlst        -
[-        ] qua…y_assessment:plasmidfinder -
[-        ] min_hash:mash_dist             -
[-        ] ave…eotide_identity:references -
[-        ] ave…ucleotide_identity:fastani -
[-        ] information:amrfinderplus      -
[-        ] information:drprg              -
[-        ] information:emmtyper           -
[-        ] information:kaptive            -
[-        ] information:kleborate          -
[-        ] information:elgato             -
[-        ] information:mykrobe            -
[-        ] information:pbptyper           -
[-        ] information:seqsero2           -
[-        ] information:serotypefinder     -
[-        ] information:shigatyper         -
[-        ] information:json_convert       -
[-        ] phylogenetic_analysis:prokka   -
[-        ] phylogenetic_analysis:panaroo  -
Plus 7 more processes waiting for tasks…
Pipeline completed at: 2024-09-11T06:46:22.350361222Z
MultiQC report can be found at grandeur/multiqc/multiqc_report.html
Summary can be found at grandeur/grandeur_summary.tsv
Execution status: failed
ERROR ~ Execution aborted due to an unexpected error

 -- Check script '/home/becksts/.nextflow/assets/UPHL-BioNGS/Grandeur/./subworkflows/../modules/local/prokka.nf' at line: 7 or see '.nextflow.log' file for more details


@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

I reran without the iqtree2_outgroup param, and got this error:
nxf.log.txt

@erinyoung
Copy link
Member

It looks like there's something wrong with your Grandeur Ref container. What happens if you delete your quay.io-uphl-grandeur_ref-2024-06-26.simg file and redownload it?

Sep-11 18:46:23.521 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: average_nucleotide_identity:references (Preparing references); status: COMPLETED; exit: 1; error: -; workDir: /data/Sequence_analysis/Grandeur/Analyses/Honor_Health_Osborne_6Sep2024/work/02/5a3247346197e3dee0fcdee8bb33ec]
Sep-11 18:46:23.527 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=average_nucleotide_identity:references (Preparing references); work-dir=/data/Sequence_analysis/Grandeur/Analyses/Honor_Health_Osborne_6Sep2024/work/02/5a3247346197e3dee0fcdee8bb33ec
  error [nextflow.exception.ProcessFailedException]: Process `average_nucleotide_identity:references (Preparing references)` terminated with an error exit status (1)
Sep-11 18:46:23.534 [TaskFinalizer-1] INFO  nextflow.processor.TaskProcessor - [02/5a3247] NOTE: Process `average_nucleotide_identity:references (Preparing references)` terminated with an error exit status (1) -- Execution is retried (1)
Sep-11 18:46:23.541 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

OK. It was from the older server. Did you mean .img? And do I use nextflow pull?

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

Error with nextflow pull;
WARN: Cannot read project manifest -- Cause: Remote resource not found: https://api.github.com/repos/nextflow-io/quay.io-uphl-grandeur_ref-2024-06-26.img/contents/nextflow.config
Remote resource not found: https://api.github.com/repos/nextflow-io/quay.io-uphl-grandeur_ref-2024-06-26.img/contents/main.nf

@erinyoung
Copy link
Member

!!!

It's missing

!!!

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

Was this the correct command?
nextflow pull quay.io-uphl-grandeur_ref-2024-06-26.img

@erinyoung
Copy link
Member

It would be something like singularity pull <image location> --name <file destination>

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

I found the "image" here:
https://quay.io/repository/uphl/grandeur_ref?tab=tags

But the size is 0 for the "latest" and "2024-06-26" tags.
How can I download it?

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 11, 2024

I chose to pull the latest image:
I ran the following:
docker pull quay.io/uphl/grandeur_ref@sha256:6149fedf329f25763e4aa561236bff923d77d590f8325099f0927034f490bbf6

It finished. It didn't place the image in /data/nextflow_cachedir. How do I find/test it?

@erinyoung
Copy link
Member

That's a docker image. Did you want to convert it to a singularity image or just use Docker for the workflow?

@DrB-S
Copy link
Contributor Author

DrB-S commented Sep 12, 2024

I would prefer converting to a singularity image. What would the switch to docker entail? I get the same error when I call "-profile docker,msa":
/data/Sequence_analysis/nextflow run -qs 10 UPHL-BioNGS/Grandeur -profile docker,msa --outdir grandeur

N E X T F L O W ~ version 24.04.4

Launching https://github.com/UPHL-BioNGS/Grandeur [stoic_allen] DSL2 - revision: 35b867a [main]

/^^^^ /^^^^^^^ /^ /^^^ /^^ /^^^^^ /^^^^^^^^ /^^ /^^ /^^^^^^^
/^ /^^ /^^ /^^ /^ ^^ /^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^
/^^ /^^ /^^ /^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^
/^^ /^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^^^^^ /^^ /^^ /^ /^^
/^^ /^^^^ /^^ /^^ /^^^^^^ /^^ /^^ /^ /^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^
/^^ /^ /^^ /^^ /^^ /^^ /^^ /^ ^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^
/^^^^^ /^^ /^^ /^^ /^^ /^^ /^^ /^^^^^ /^^^^^^^^ /^^^^^ /^^ /^^
Currently using the Grandeur workflow for use with microbial sequencing.
The view is great from 8299 feet (2530 meters) above sea level.

Author: Erin Young
email: [email protected]
Version: 4.5.24250
The files and directory for results is grandeur
executor > local (1)
[- ] quality_assessment:quast -
[- ] quality_assessment:mlst -
[- ] quality_assessment:plasmidfinder -
[- ] min_hash:mash_dist -
[ff/f7c5e8] average_nucleotide_identity:references (Preparing references) [100%] 1 of 1 ✔
[- ] average_nucleotide_identity:fastani -
[- ] information:amrfinderplus -
executor > local (1)
[- ] quality_assessment:quast -
[- ] quality_assessment:mlst -
[- ] quality_assessment:plasmidfinder -
[- ] min_hash:mash_dist -
[ff/f7c5e8] average_nucleotide_identity:references (Preparing references) [100%] 1 of 1 ✔
[- ] average_nucleotide_identity:fastani -
[- ] information:amrfinderplus -
[- ] information:drprg -
[- ] information:emmtyper -
[- ] information:kaptive -
[- ] information:kleborate -
[- ] information:elgato -
[- ] information:mykrobe -
[- ] information:pbptyper -
[- ] information:seqsero2 -
[- ] information:serotypefinder -
[- ] information:shigatyper -
[- ] information:json_convert -
[- ] phylogenetic_analysis:prokka -
[- ] phylogenetic_analysis:panaroo -
[- ] phylogenetic_analysis:core_genome_evaluation -
Plus 10 more processes waiting for tasks…
ERROR ~ Execution aborted due to an unexpected error

-- Check script '/home/becksts/.nextflow/assets/UPHL-BioNGS/Grandeur/./subworkflows/../modules/local/prokka.nf' at line: 7 or see '.nextflow.log' file for more details

@erinyoung
Copy link
Member

My apologies for taking so long to get back to you!!!

It might be your version of singularity, but this is one issue that I've seen before.

Nextflow will pull your singularity images and store them in your singularity cache directory.

If not, this is how you create a singularity image using both docker and singularity.

docker pull quay.io/uphl/grandeur_ref:2024-06-26
docker save quay.io/uphl/grandeur_ref:2024-06-26 -o grandeur_ref_image.tar
singularity build /data/nextflow_cachedir/quay.io-uphl-grandeur_ref-2024-06-26.img docker-archive://grandeur_ref_image.tar

This specific image will need to be named "quay.io-uphl-grandeur_ref-2024-06-26.img" and put in your nextflow singularity cachedir, which I think is /data/nextflow_cachedir.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants