vcycle.conf.5

.TH vcycle.conf 5 "Apr 2018" "vcycle.conf" "vcycle Manual"
.SH NAME
.B vcycle.conf
\- vcycle configuration file
.SH DESCRIPTION
.B vcycled
is a daemon  which implements VM lifecycle management on IaaS cloud systems
in a way inspired by Vac. vcycled reads its configuration from
.B /etc/vcycle.conf
and .conf files in
.B /etc/vcycle.d

The configuration files use the Python ConfigParser syntax, which is similar
to MS Windows INI files. The files are divided into sections, with each section
name in square brackets. Each section contains
a series of option=value pairs. Sections with the same name are merged
and if options are duplicated, later values overwrite values given
earlier.

For ease of management, any configuration file ending in .conf in the
directory /etc/vcycle.d will be read, in
alphanumeric order by name, and then /etc/vcycle.conf is read if present.

.SH [SPACE ...] SECTIONS

One [space ...] section must exist for each project, tenancy, or account in which
VMs will be created, with the Vcycle space name for the space given in the section
name, such as [space project1.example.com]. The Vcycle admin has a free choice
of what names to give each space, but the name must only consist of lowercase
letters, numbers, periods, and hyphens so it can be used as the DNS name of
a virtual CE elsewhere in the system.

.B api
is required and specifies the lowercase name of the API to use when contacting
the cloud service associted with this space. API names can consist of letters,
numbers, and underscores.

.B api_version
is optional and requests a particular version of the API understood by its
plugin. API versions can consist of lowercase letters, numbers, hyphens,
periods, and underscores.

.B processors_limit
prevents new machines from being created if the number of processors for
this space in any state exceeds the given limit.

.B flavor_names
gives a space-separated list of one or more names of flavors defined
by the cloud system, which may be used by machinetypes within this space.

.B gocdb_sitename
gives the GOCDB site name to use when writing APEL
accounting record files to /var/lib/vcycle/apel-outgoing and
/var/lib/vcycle/apel-archive. Please use your official site name to avoid
the risk of misnamed records getting into the central APEL database.
If gocdb_sitename is not given, then records are only written to
apel-archive and the Vcycle space name is used as a placeholder in the
files.

.B gocdb_cert_file
and
.B gocdb_key_file
are the absolute paths of an X.509 certificate and private key which
are authorized to update the GOCDB capacity and VO information for this
space. The space must already exist in GOCDB as part of the site with name
given by 
.B gocdb_sitename.
The list of VOs with machinetypes in this space is taken from the Vcycle
configuration with FQANs given by
.B accounting_fqan.
Both machinetypes defined locally and those from remote vacuum pipes are
included. Updates are sent to GOCDB once every 24 hours.

.B vacmon_hostport
If set, this option gives a space-separated list of HOST:PORT to send
VacQuery UDP messages to. This can be used to monitor the ongoing status
of the Vcycle factory and its VMs via site or central VacMon services.
The central GridPP VacMon service is vacmon.gridpp.ac.uk:8884

.B https_port
gives the port number used to contact the Vcycle HTTPS server from
within the VMs. This only needs to be changed if there is an intervening
firewall. Default 443.

.B shutdown_time
gives a time to set shutdowntime_job files to, in Unix time. By default these
are just set to be time + max_wallclock_seconds. If this time is in the past,
vcycle stops creating machines for the space.

.SH OPENSTACK SPACE SECTIONS

OpenStack spaces are enabled with
.B api = openstack
and the following options are then available.

.B glance_api
is the glance api version.

.B url
is the URL of the identity (KeyStone) endpoint for this OpenStack service.

.B project_name
is the project or tenancy name.

.B domain_name
is the domain name containing the project or 'default' by default.

.B region
and
.B zones
optionally give the OpenStack region and zone(s) to use when creating
VMs.

.B network_uuid
gives the UUID of a network to use. By default no network is requested
and OpenStack uses the default network for the project.

.B username
is a user with access to the project.

.B password_base64
is the base64 encoded password for the user. Vcycle will decode the
value to obtain the true password. This encoding is to avoid revealing
memorable passwords when casually viewing or editing configuration files.
You can encode the real password with something like:
.br
.ce
echo -n 'PASSword' | base64

When creating VMs in OpenStack spaces, Vcycle will create "machinefeatures",
"jobfeatures", and "joboutputs" metadata keys with the URLs of the
corresponding directories for the VM on the Vcycle machine's HTTP(S)
server.

.B os_shutdown_timeout
sets the "os_shutdown_timeout" image metadata tag, which overrides the time
given to instances to shutdown gracefully. Currently sets all for a given
space.

.SH EC2 SPACE SECTIONS

As well as OpenStack, Vcycle natively supports the EC2 API. However, you
should use the OpenStack API if available. Spaces accessed with the EC2 API
are enabled with
.B api = ec2
and the following options are then available.

.B url
is the URL of the endpoint of this EC2 service.

.B access_key
and
.B secret_key
are the credentials needed to access the service.

.B version
explicitly gives the API version to be passed to EC2. Default "2010-08-31".

.B region
explicitly gives the region to be passed to EC2. Default "us-east-1".

.B service
explicitly gives the service name to be passed to EC2. Default "openstack".

When creating VMs in EC2 spaces, Vcycle will create "machinefeatures",
"jobfeatures", and "joboutputs" metadata tags with the URLs of the
corresponding directories for the VM on the Vcycle machine's HTTP(S)
server.

.SH MACHINE / JOB FEATURES HTTP(S) SERVER

Vcycle will create the machinefeatures and jobfeatures files according
to the WLCG MJF protocol described in HSF-TN-2016-02 in machinefeatures and
jobfeatures subdirectories
of the VMs directory in /var/lib/vcycle/machines . Using the vcycle.httpd.inc
Apache configuration file supplied in /usr/share/doc/vcycle-VERSION, these
files are publish via HTTP and HTTPS.

In addition, if x509dn is set for the VM's machinetype, then the script vcycle-cgi
will allow the VM to write heartbeat, logging, and shutdown_message files
to the VM's $JOBOUTPUTS directory.

This server also publishes the heartbeat machines lists of the machines with
a current heartbeat for each machinetype in each space described in the
.B USER_DATA SUBSTITUTIONS
section.

.SH [MACHINETYPE ... ...] SECTIONS

One [machinetype ... ...] section must exist for each machinetype in each space, with
the space name and name of the machinetype given in the section name, such as
[machinetype project1.example.com example].
A machinetype name must only consist of lowercase letters, numbers, and hyphens.
Each of these sections contain option=value pairs that are specific to
that machinetype. The same machinetype name can appear in different spaces and will
be managed separately.

.B flavor_names
gives a space-separated list of one or more names of flavors defined
by the cloud system which represent particular combinations of CPU, memory, and
disk geometry. This option overrides any flavors list given in the space
section. Subject to any constraints imposed by
.B min_processors
and
.B max_processors
the first allowed flavor in the list is used.

.B min_processors 
and
.B max_processors
give the minimum and maximum number of logical processors which can be allocated
to LMs of this type when they are created. Vcycle will attempt to select a 
flavor from the flavor_name list matching these constraints. If the API plugin
cannot determine the number of processors allocated to a VM, then Vcycle will use 
.B min_processors
in its calculations and limits, and when setting the values of 
$MACHINEFEATURES/total_cpu and $JOBFEATURES/allocated_cpu supplied to the VM.
The default minimum is 1 and the default is no maximum.

.B hs06_per_processor
gives the HEPSPEC06 power of each processor in the virtual machines created
for this flavor in this machinetype. If set, this is used to calculate the value
$MACHINEFEATURES/hs06 and $JOBFEATURES/hs06_job
supplied to the VM. It is also used when calculating target shares and
when writing APEL accounting records, and for both of these a default of 1.0
is used if not set explicitly.

.B mb_per_processor
gives the number MB of each virtual machine created for this flavor
in this machinetype. This is used as the value of $JOBFEATURES/max_rss_bytes
supplied to the VM. If the api plugin can
positively determine the number from metadata about the flavor, it will be
used in preference to the value given here. Default 2048 per processor.

.B target_share
gives the desired share of the capacity available in this space for this
machinetype. The shares do not need to add up to 1.0, and if a share is not given
for a machinetype, then it is set to 0. Vcycle consults these shares
when deciding which machinetype to start as VM capacity becomes available.
Shares are weighted by the hs06 value of the machinetype.

.B backoff_seconds
is the delay after a VM of this machinetype aborts. If a VM aborts, then no new
VMs of this type will be created for this amount of time. This can be used
to prevent the unnecessary creation of many VMs when no work is available,
and avoid overloading the matcher or task queue of the VO.

.B fizzle_seconds
is used in three places within the backoff procedure and in two
other parts of Vcycle:
.br
(1) First, if a VM finishes
without producing a shutdown message code and has lasted less than
fizzle_seconds, then it is treated as aborted.
.br
(2) Secondly, after the
backoff_seconds time has expired for a VM abort, once at least one VM has
been started in this Vcycle space, then no more new VMs can be started for
another fizzle_seconds.
.br
(3) Additionally, when writing the accounting log files, any VMs which
run for less than fizzle_seconds are excluded.
.br
(5) Finally, the heartbeat file
checking is only carried out once an initial period of fizzle_seconds
has passed.

.B accounting_fqan
is used to specify a FQAN to include when writing APEL accounting
records, to associate usage with particular experiments.

.B processors_limit
prevents new machines from being created if the number of processors for
this machinetype in any state exceeds the given limit.

.B max_wallclock_seconds
gives the maximum lifetime of a VM. Vcycle will create
$MACHINEFEATURES/shutdowntime inside the VM using this value to
communicate it to the VM. Vcycle will destroy the VM if it is still
running after this amount of time. Default 86400.

.B heartbeat_file
allows the machinetype to nominate a file which will be created in
the $JOBOUTPUTS directory before fizzle_seconds has passed. If this
file is not created by then and maintained for the lifetime of the VM,
the VM will be destroyed.

.B heartbeat_seconds
gives the frequency at which the heartbeat_file must be updated after
fizzle_seconds has passed. If the file is not updated for
heartbeat_seconds then the VM will be destroyed. If heartbeat_seconds
is 0, then only the existence of the file will be checked. Default 0.

.B cvmfs_proxy_machinetype
gives the name of another machinetype consisting of HTTP caching proxies
suitable for cvmfs running inside the VMs. The list of proxies is made
available to VMs via the ##user_data_option_cvmfs_proxy## substitution.
If :PORTNUMBER is appended to the name of the machinetype, then that 
port number will be used in the cvmfs proxy lists Vcycle generates.
See 
.B USER_DATA SUBSTITUTIONS
for more details.

.B x509dn
is an optional X.509 DN which will be used by the vcycle-cgi script to
control writing to VMs' $JOBOUTPUTS directories on the local HTTPS
server.

.B log_joboutputs
can be set to True to enable recording of all the files from
local $JOBOUTPUTS directories for VMs, to subdirectories of
/var/lib/vcycle/joboutputs when the VMs finish or are killed. The
subdirectories are in a hierarchy of the space name, machinetype name,
and then hostname of the VM. Default False.

.B joboutputs_days
sets the expiration time in days for per-VM directories created under
/var/lib/vcycle/joboutputs.

.B remote_joboutputs_url
sets a base URL on a remote HTTPS server to which VMs of this machinetype
can write. The value of $JOBOUTPUTS will be the VM
name chosen by Vcycle appended as a directory name to the URL given
by this option.

.B legacy_proxy
can be set to True to generate Globus legacy proxies rather than RFC 3820
proxies. Default False.

.B user_data_proxy
set to true causes the files x509cert.pem and x509key.pem in the
machinetype's subdirectory of /var/lib/vcycle/spaces/SPACE/machinetypes to
be used to make a limited X.509 proxy. The two files can be
identical if desired, and the X.509 certificate and RSA private key
will be extracted from the files as appropriate. (Note that this location
is one level about the files subdirectory in which the following options
look by default.)

For the remaining options, if the file name begins with '/', then it
will be used as an absolute path; otherwise the path will be interpreted
relative to the machinetype's subdirectory of /var/lib/vcycle/spaces/SPACE/machinetypes/MACHINETYPE/files
where SPACE is the parent space name and MACHINETYPE is the name of
this machinetype.

.B remote_joboutputs_cert
and
.B remote_joboutputs_key
give filesnames of an X.509 client
certificate and key to use when requesting
$JOBOUTPUTS/shutdown_message and any heartbeat file in $JOBOUTPUTS. If
both are contained in the same file then the same value can be given
to both options.

.B root_image
identifies the image file from which the VM will boot. If the cloud
service already has the desired image, then it can be referenced by
prefixing the service's native image ID with "image:".
.br
For the OpenStack API, root_image can be
the path to the image file itself on the local filesystem. Alternatively,
it can also be a remote HTTP or HTTPS URL which Vcycle
will cache in /var/lib/vcycle/imagecache. The remote server must supply a
Last-Modified timestamp and Vcycle will re-request the image each time a
VM starts using an If-Modified-Since request to minimise network load.
Alternatively, the images may be files in the local filesystem. If
root_image ends in .iso , then the image will be declared as ISO format
(a CD-ROM image), otherwise as a raw HDD image.

.B cernvm_signing_dn
is used to specify a regular expression to match the DN of an X.509
certificate used to verify the authenticity of the root image. Vcycle
attempts to obtain the certificate and signature from a CernVM Signature
Block at the end of the image file, verifies the
certificate using the CA files in /etc/grid-security/certificates, and
compares the certificate DN to cernvm_signing_dn. If this option is
given, all these verification steps must be satisified for the image
to be used. As of 2016, CernVM images are signed with a DN matching
the regular expression /CN=cvm-sign01\\.cern\\.ch$

.B root_public_key
is the file name of a public key which Vcycle will set up on the cloud
system and supply to the VMs to allow root ssh access. Setting this
option to /root/.ssh/id_rsa.pub will give access from the factory machine.

.B user_data
is the path of a contextualization file provided by the VO and perhaps
modified by the site. If the path is a remote HTTP or HTTPS URL, Vcycle
will fetch it over the network each time a VM is started. However the
file is obtained, Vcycle will apply a series of default and locally defined
##user_data___## substitutions to it. See USER_DATA SUBSTITUTIONS below
for a list of the default substitutions.

.B user_data_option_XXX
and
.B user_data_file_XXX
are locally defined substitutions which will be applied to the user_data
file before the VM is started. user_data_option_XXX takes the string to
be substituted. user_data_file_XXX takes the relative or absolute path to
a file whose contents will be substituted for the pattern in the
user_data file.

.SH USER_DATA SUBSTITUTIONS

Before the user_data file is used in starting a VM, several pattern based
substitutions are performed by Vcycle. These patterns are in the form
##user_data___##. String values given to the option user_data_option_XXX
replace patterns of the form ##user_data_option_XXX##. The contents of
the files given to user_data_file_XXX options also replace patterns of the
form ##user_data_option_XXX##. In both cases XXX are arbitrary strings
consisting of letters, numbers, and underscores.

The pattern ##user_data_x509_proxy## is replaced by the X.509 proxy 
created if the user_data_proxy_cert and user_data_proxy_key options
are given.

.B cvmfs_proxy_machinetype
may be given in a machinetype definition 
with the name of another machinetype consisting of HTTP 
caching proxies suitable for cvmfs. The list of proxies is made
available to VMs via the normal ##user_data_option_cvmfs_proxy## 
substitution.
Only machines with a valid heartbeat are included, and they are 
identified by IP address in the format http://xxx.xxx.xxx.xxx:ppp and 
separated by pipe ('|')
characters so they are used in round-robin mode by cvmfs. For
security reasons, these proxies are accessed on port 280 by default 
not 3128. This port number may be changed by appending :PORTNUMBER 
to the given machinetype name. 
If user_data_option_cvmfs_proxy is also given, then a semicolon
separator and the option's value are appended to
the list of proxies obtained from the machinetype's list. This allows
a static choice of backup proxies to be given, which will benefit from
cvmfs's failover feature if none of the machinetype proxies are usable.

In addition, the following substitutions are performed automatically by
Vcycle using data it holds internally:

.br
.B ##user_data_space##
is the Vcycle space name.
.br
.B ##user_data_url##
is the HTTP(S) URL from which the user_data template was obtained. Only given if
the template was retrieved by HTTP(S) rather from a local path.
.br
.B ##user_data_machinefeatures_url##
and
.B ##user_data_jobfeatures_url##
and
.B ##user_data_joboutputs_url##
are the values of $MACHINEFEATURES, $JOBFEATURES, and $JOBOUTPUTS to set
within the VM.
.br
.B ##user_data_heartbeat_machines_url##
is the HTTP(S) URL from which a list of the machines in this space with
the same machinetype as this machine can be found. Only machines with a
current heartbeat are included. The list consists of the heartbeat time
in Unix seconds, the machine name, and the local IP address of the machine 
within the remote cloud.
.br
.B ##user_data_machinetype##
is the name of the machinetype of this VM.
.br
.B ##user_data_machine_hostname##
is the hostname given to the VM by Vcycle.
.br
.B ##user_data_manager_version##
has the form "Vcycle v.v.v" where v.v.v is the Vcycle version.
.br
.B ##user_data_manager_hostname##
is the hostname of the machine on which the Vcycle daemon is running.

.SH AUTHOR
Andrew McNab <Andrew.McNab@cern.ch>

vcycled is part of Vcycle: https://www.gridpp.ac.uk/vcycle/
.SH "SEE ALSO"
.BR vcycled(8),
.BR vcycle-cgi(8)