-
Notifications
You must be signed in to change notification settings - Fork 2
Home
Welcome to the Washington University CCF HCP Pipelines Singularity wiki!
The HCP Pipelines Singularity repository is for files to help run the HCP Pipelines using Singularity.
This is the place for documentation for using the HCP Pipelines Singularity repository.
Singularity is a container platform specifically designed for scientific and high performance computing (HPC) use cases.
- Installing Singularity
- Getting the HCP Pipelines Singularity repository
- Modifying the Singularity recipe file
- Building a Singularity image file to run HCP Pipelines
- Working with your Singularity image file
Back to Table of Contents
See Singularity Quick Start Installation for further information.
The following steps have been testing on Ubuntu 16.04 LTS
$ sudo apt-get install build-essential libarchive-dev
$ wget https://github.com/singularityware/singularity/releases/download/2.5.2/singularity-2.5.2.tar.gz
$ tar xvf singularity-2.5.2.tar.gz
$ cd singularity-2.5.2
$ ./autogen.sh
$ ./configure --prefix=/usr/local
$ make
$ sudo make install
$ which singularity
/usr/local/bin/singularity
$ singularity --version
2.5.2-dist
$ sudo yum groupinstall "Development Tools"
$ sudo yum install libarchive-devel
$ wget https://github.com/singularityware/singularity/releases/download/2.5.2/singularity-2.5.2.tar.gz
$ tar xvf singularity-2.5.2.tar.gz
$ cd singularity-2.5.2
$ ./autogen.sh
$ ./configure --prefix=/usr/local
$ make
$ sudo make install
$ which singularity
/usr/local/bin/singularity
$ singularity --version
2.5.2-dist
Back to Table of Contents
If you have git installed on your system, you can simply clone the HCP Pipelines Singulary repository as follows:
git clone https://github.com/Washington-University/HCPpipelinesSingularity.git
If you don't have or don't care to use git, then you can use the following technique to get a copy of the current latest code in the master branch of the repository. This will not allow
you to interact with the repository (e.g. check out branches, make and commit modifications, etc.) via git.
$ wget https://github.com/Washington-University/HCPpipelinesSingularity/archive/master.tar.gz
$ tar xvf master.tar.gz
HCPpipelinesSingularity-master/
HCPpipelinesSingularity-master/.gitignore
HCPpipelinesSingularity-master/HCPprocessPipelines
HCPpipelinesSingularity-master/LICENSE.md
HCPpipelinesSingularity-master/README.md
HCPpipelinesSingularity-master/product.txt
HCPpipelinesSingularity-master/show_version
HCPpipelinesSingularity-master/version.txt
$ mv HCPpipelinesSingularity-master HCPpipelinesSingularity
Back to Table of Contents
There are some files that are required in order to run the HCP Pipelines that cannot be distributed as part of a Singularity container or automatically retrieved from a known distribution location when the Singularity container is created from a recipe file. You will have to get or generate those files yourself, place them in a known location on your system before building a Singularity container image from a recipe, and modify the Singularity recipe file to reflect the location of those files.
Gradient Distortion Coefficients file
The first of these files that cannot be distributed is the file that contains the gradient distortion coefficients for your scanner. The information in that file is considered to be proprietary information owned by the scanner manufacturer. Therefore, we cannot provide that information as part of an already created Singularity container.
Also, because this is proprietary information, gradient coefficients files cannot be made available for general download from a known site. Therefore, we cannot simply put a reference to a known download site in the Sigularity recipe file.
You will need to obtain the gradient distortion coefficients file either from your scanner
manufacturer or from the scanner control computer itself. For example, on Siemens scanners,
the gradient coefficients file is usually available at C:\MedCom\MriSiteData\GradientCoil\coeff.grad.
As it is currently written, the Singularity recipe file in the HCP Pipelines Singularity
repository, HCPprocessPipelines, assumes that you will create a directory somewhere on your system
and place any number of gradient coefficients files in that directory. This is so you can build one
Singularity container from the recipe file that supports running HCP Pipelines on data collected from
a number of different scanners.
Use the following steps to incorporate your gradient coefficients file(s) into the Singularity container that you build.
-
Create a directory on the system on which you will build the Singularity container (e.g.
/home/tbb/gradient_coefficients_files).Note that the system on which you will build the Singularity container does not need to be the same one on which you actually run the processing.
-
Copy your gradient distortion coefficients files into that directory.
e.g.
cp *.grad /home/tbb/gradient_coefficients_files -
Edit the
HCPprocessPipelinesSingularity recipe file in your local copy of the HCP Pipelines Singularity repository.- Find the
%filessection in the Singularity recipe file. It will look something like:
- Find the
%files ########## Need gradient_coefficient_files!!! && freesurfer license key file from https://surfer.nmr.mgh.harvard.edu/fswiki/License
/home/j/Downloads/xnat_pbs_jobs_container/gradient_coefficient_files/* /export/HCP/gradient_coefficient_files/
/home/j/Downloads/xnat_pbs_jobs_container/license.txt /export/freesurfer_license/
-
Change the
/home/j/Downloads/xnat_pbs_jobs_container/gradient_coefficient_files/*to point to where you have placed your gradient coefficients files on your system.That is, substitute the location in which your gradient coefficients files reside for
/home/j/Downloads/xnat_pbs_jobs_container/gradient_coefficient_files/*For example:
/home/tbb/gradient_coefficients_files/*
For example:
/home/tbb/gradient_coefficients_files/* /export/HCP/gradient_coefficient_files/
FreeSurfer License file
Another file which we can not build in to a Singularity container for distribution nor specify a known download location into a Singularity recipe file is a FreeSurfer license file. This is a simple text file that you must obtain by visiting https://surfer.nmr.mgh.harvard.edu/fswiki/License.
On that page, you will find a link labeled "Follow this link to obtain a license key". Follow the link
as instructed, and fill out the form with the requested information. When asked for your
Operating System/Platform:, be sure to select Linux/Intel. The container will be effectively
running CentOS/Linux, so you will be using the Linux version of FreeSurfer.
Once you have filled out the form, select the I AGREE button to agree to the FreeSurfer license.
You will be sent a FreeSurfer license file via email. The FreeSurfer license file is a simple
text file attached to the email sent to you. It will likely be named license.txt.
Save the license.txt file to a directory on your system (e.g. /home/tbb/FreeSurferLicense/license.txt).
In the %files section of the Singularity recipe file, change the line that reads
/home/j/Downloads/xnat_pbs_jobs_container/license.txt /export/freesurfer_license/.
Substitute your license file location for /home/j/Downloads/xnat_pbs_jobs_container/license.txt
For example:
/home/tbb/FreeSurferLicense/license.txt /export/freesurfer_license/
Back to Table of Contents
Save the above modifications to the HCPprocessPipelines Singularity recipe file, and then
issue the following command while in the directory containing the HCPprocessPipelines Singularity
recipe file to create the container image. Note, that you must either be the root user
or be able to use the sudo command in order to build a Singularity container image from a
Singularity recipe file.
$ sudo singularity build HCPprocessPipelines.img HCPprocessPipelines
There should be quite a lot of text feedback from the singularity command as it builds
the Singularity container image based on the recipe file. This process is analogous to
installing an operating system (CentOS 7) and all the required software for running
the HCP Pipelines on a new computer. This process can take an hour or more.
Fairly early in the textual feedback, you may see some warning messages similar to the following:
WARNING: Warning reading tar header: Ignoring malformed pax extended attribute
WARNING: Warning reading tar header: Ignoring malformed pax extended attribute
WARNING: Warning reading tar header: Ignoring malformed pax extended attribute
Note that these warnings do not affect the output. See https://github.com/singularityware/singularity/issues/1669 for further information.
Back to Table of Contents
$ singularity apps HCPprocessPipelines.img
FunctionalPreprocessing
FunctionalPreprocessingCheck
StructuralPreprocessing
StructuralPreprocessingCheck
$ singularity help --app <app-name> HCPprocessPipelines.img
e.g.
$ singularity help --app StructuralPreprocessing HCPprocessPipelines.img
$ singularity shell HCPprocessPipelines.img ... Singularity HCPprocessPipelines.img:~/projects/HCPpipelinesSingularity> cd /export/freesurfer-6.0 Singularity HCPprocessPipelines.img:/export/freesurfer-6.0> ls ... This is where FreeSurfer 6.0 is installed on the container "machine". ... Singularity HCPprocessPipelines.img:/export/freesurfer-6.0> cat license.txt ... This should show the contents of your FreeSurfer 6.0 license file. ... Singularity HCPprocessPipelines.img:/export/freesurfer-6.0> cd /export/HCP/Pipelines Singularity HCPprocessPipelines.img:/export/HCP/Pipelines> ls ... This is the version of the HCP Pipelines that you are using in your container ... Singularity HCPprocessPipelines.img:/export/HCP/Pipelines> cd global/config Singularity HCPprocessPipelines.img:/export/HCP/Pipelines/global/config> ls ... This should show your gradient coefficient files along with other HCP Pipelines global configuration files ... Singularity HCPprocessPipelines.img:/export/HCP/Pipelines/global/config> cd /pipeline_tools/xnat_pbs_jobs Singularity HCPprocessPipelines.img:/pipeline_tools/xnat_pbs_jobs> ls
Back to Table of Contents