On torque cluster, piped_bamprep "result not ready, cannot allocate memory" error #3713
Comments
|
We ran it on slurm a lot, but not on torque, so there might be unsolved issues. You could try to increase the timeout and controller job memory (-r conmem=2) However, in your case it is easier to run one node bcbio projects/sample without ipython (small gene panel). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi Folks
I am having trouble getting bcbio to run on my torque cluster with variant2.
I updated to the latest development release, but it was giving the same problem before.
The cluster has 10x nodes with 28 cores and 128 GB RAM each.
The analysis is of a Trusight cancer panel sample, Illumina PE, sequencing 96 genes.
The ref here is still hg19 for comparison to previous results.
When I run the job on a single machine, everything is fine.
When I run the job over multiple nodes using torque with:
bcbio_nextgen.py ../config/omega.yaml -t ipython -n 112 -s torque -q long
it get past bwa mapping, and then seems to die on piped_bamprep as below.
Here is the relevant part of my bcbio_system.yaml file:
resources:
default options, used if other items below are not present
avoids needing to configure/adjust for every program
default:
memory: 4G
cores: 28
jvm_opts: ["-Xms4000m", "-Xmx4000m"]
Can also set program specific options
gatk:
jvm_opts: ["-Xms4000m", "-Xmx4000m"]
snpeff:
jvm_opts: ["-Xms4000m", "-Xmx4000m"]
qualimap:
memory: 4g
express:
memory: 8g
dexseq:
memory: 8g
macs2:
memory: 18g
seqcluster:
memory: 8g
(PS For jvm_opts, is the memory also set per core?)
All config and log files are appended below.
The error is:
...
...
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: calculate_sv_bins
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: calculate_sv_coverage
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: normalize_sv_coverage
[2023-06-27T14:36Z] wonko.bi.up.ac.za: Timing: hla typing
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: call_hla
[2023-06-27T14:36Z] wonko.bi.up.ac.za: Resource requests: gatk, gatk-haplotype, picard; memory: 4.00, 4.00, 4.00; cores: 1, 28, 28
[2023-06-27T14:36Z] wonko.bi.up.ac.za: Configuring 79 jobs to run, using 1 cores each with 4.00g of memory reserved for each job
[2023-06-27T14:37Z] wonko.bi.up.ac.za: Timing: alignment post-processing
[2023-06-27T14:37Z] wonko.bi.up.ac.za: ipython: piped_bamprep
[2023-06-27T14:37Z] wonko.bi.up.ac.za: Did not stop IPython cluster correctly
Traceback (most recent call last):
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 689, in _ordered_iter
rlist = self.get(0)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 171, in get
raise error.TimeoutError("Result not ready.")
ipyparallel.error.TimeoutError: Result not ready.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/cluster_helper/cluster.py", line 1151, in cluster_view
yield cluster_view.view
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/prun.py", line 56, in start
yield ipython.runner(view, parallel, dirs, config)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/pipeline/main.py", line 152, in variant2pipeline
samples = region.parallel_prep_region(samples, run_parallel)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/pipeline/region.py", line 164, in parallel_prep_region
"piped_bamprep", _add_combine_info, file_key, ["config"])
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/split.py", line 62, in parallel_split_combine
split_output = parallel_fn(parallel_name, split_args)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/ipython.py", line 137, in run
for data in view.map_sync(fn, items, track=False):
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 670, in iter
for r in it():
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 695, in _ordered_iter
for r in self._yield_child_results(child):
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 681, in _yield_child_results
error.collect_exceptions(rlist, self._fname)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/error.py", line 233, in collect_exceptions
raise e
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/error.py", line 231, in collect_exceptions
raise CompositeError(msg, elist)
ipyparallel.error.CompositeError: one or more exceptions from call to method: piped_bamprep
[53:apply]: ImportError: /apps/bcbio/anaconda/lib/python3.7/site-packages/numpy/fft/_pocketfft_internal.cpython-37m-x86_64-linux-gnu.so: failed to map segment from shared object: Cannot allocate memory
The system seems to find the node memory more-or-less correctly:
provenance/system-ipython-long.yaml shows:
memory: 3023.428493499756
name: wonko1.bi.up.ac.za
Any help with this problem would be very highly appreciated!
Best regards!
Fourie
Version info
To Reproduce
bcbio_nextgen.py ../config/omega.yaml -t ipython -n 112 -s torque -q longRunning bcbio version: 1.2.9
Your yaml configuration file:
details:
tools_off: peddy
align_split_size: 5000000
aligner: bwa
jointcaller: gatk-haplotype-joint
mark_duplicates: true
nomap_split_targets: 100
platform: illumina
quality_format: standard
realign: false
recalibrate: false
remove_lcr: true
variant_regions: /nlustre/users/dewalde/OmegaAll/bcbio/trusight_cancer_manifest_a.bed
variantcaller: gatk-haplotype
analysis: variant2
description: BRB101
files:
genome_build: hg19
metadata:
batch: omega_all
phenotype: breast_cancer
sex: female
fc_date: '2018-03-15'
fc_name: omega
upload:
dir: ../final```
The text was updated successfully, but these errors were encountered: