NEWS

Release notes for OpenLava versions

Master
========

-- Implement SLOTS_PER_PROCESSOR in lsb.resources.

4.0beta2
============

-- Add Platform MPI wrapper.
-- Use bash for etc/openlava to support Ubuntu.
-- Fixed rusage string parsing bug.
-- Introduce USUSP and SSUSP counters in the fairshare accounts. Previously
   suspended jobs were accounted as running leading to erroneous state
   transition from suspended to pending state.
-- Implement a new lsf.conf variable named LSB_DEFAULT_USER_GROUP="group name".
   If this variable is specified than jobs that do not specify -G at submission
   time will be assigned LSB_DEFAULT_USER_GROUP as group. This group must be
   configured among valid groups in a fairshare queue.
-- In the fairshare scheduling if a user has jobs that enforce a user group
   and jobs that don't, schedule both types of jobs with -G jobs first.
-- Implement a set of fixes related to fairshare job selection which were
   causing the scheduler to slowly dispatch jobs even if slots were available
   and set incomplete pending reasons.
-- Fixed slot based and resource preemption job replay bug. Fixed slot based
   preemption memory leak, and man pages.
-- Implement a slot based queue preemption in which jobs are suspended,
   instead of requeued.
-- If the MBD does not set the pending reasons for a job print:
   "System is unable to set pending reason for job;" in the bjobs -p
   output instead of blank or meaningless string.
-- badmin ckconfig checks the ownership and permission of work/logdir.
-- In bqueues -l output change the keyword RAN to STARTED to reflect
   the number of started jobs and not only ran.
-- lsid prints the 4.0beta2 identification for the current branch.
-- Implement the decay slot factor for HIST_MINUTES.

4.0beta1
========
-- Fix mbatchd core dump that happens if jobs are submitted with -t option
   of bsub and the job is terminated while pending.
-- Added bjobs -o to support custom output fields
-- Added bjobs -cwd to specify the job current working directory
-- Implement MBD_DEDICATED_RESOURCES in lsf.conf. These are host based boolean
   resources that must be defined in lsf.cluster file. A job must explicitly
   specify a dedicated resource to get allocated the host with that resource.
-- Fix a typo in lsb.queues.in, preemption was spelled with double t.
-- Fix compatibility problems when replaying older lsb.events.
-- Fix the XDR error in the badmin qclose command.
-- Add man page for $LSF_ENVDIR/openlava
-- Add a template file $LSF_ENVDIR/limits.sh
-- Fixed sbatchd compile error on RHEL 5 because glibc lacks CPU_COUNT.
-- Fix a fairshare bug in which jobs were not running under the correct
   user group if bsub -G was specified.
-- Deleted warning message if mbatchd cannot find lsb.resources file.
   lsb.resources is optional.
-- Updated lsb.resources man page for supported use case.
-- Fixed binding incorrect cpus to a queue when HOSTS_SHARES is configured
   in lsb.queues.
-- Enlarge the -J option for the job name to be at least 4096 characters.
-- Added support for binding multiple slots to a core by introducing HOSTS_SHARES
   parameter in lsb.queues for specifying the number of shares the queue is
   allocated for particular host/hostgroup. The share represents the proportion of
   the total number of cores on a host that will be allocated to the queue.
-- Updated non-numa cpu binding feature to support binding multiple
   cores to a job.
-- Added slot limit per host for specific queues in lsb.resources.
-- Added lsb.params PREEMPTABLE_RESOURCES parameters. This enable resource
   preemption in MBD If this parameter is configured the queue preemption
   is done based on the specified resources and not by the host slots which is
   the default.
-- Added the blaunch.1 man page in the batch man section.
-- Developed new API ls_getrpid() to get a remote process id of a task
   started by RES knowing the local taskid.
-- Developed new API lsb_launch() to start a components of parallel jobs.
   This API also collects resource usage of the remote tasks and reports
   the to OpenLava so they can be displayed with the bjobs command.
-- Developed the blaunch command, if mpiexec/mpirun is integrated with
   blaunch, it can launch mpi parallel application, or it can launch
   home grown parallel applications. It is a wrapper around the lsb_launch()
   API.
-- In the bootstrap.sh file create the revision number file
   which is going to be installed in the etc directory.
-- Create lsid 4.0dev4 output.

4.0pre3
=======

-- Create lsid 4.0pre3 tag output.

4.0pre2
========

Introduce 4.0dev3 lsid output. The dev indicates that the code
is between tags. 4.0dev3 finishes with 4.0pre3 tag. After which
we will start 4.0dev4 lsid output till we tag 4.0pre4 or we cut
a final 4.0 branch. In this way when building tagged version we
know the exact commit.

4.0pre1
========

-- Fix res/nios hang blocking ol make.
-- Create a new parameter NIOS_MAX_TASKS specifying the maximum number
   of tasks nios can handle.
-- Fix/reimplement btop/bbot. Since the job queue management has undergone
   performance improvements these function had to be coded as well.
-- Fix the mbatchd variable MBD_NO_QSORT_JOBS, which was incorrectly
   set to MBD_QSORT_JOBS in daemons.c
-- Fix formatting bug in bparams.
-- Add badmin hclose/hopen -C "message", and fix parsing error of "badmin
   subcommand <command_options>".
-- Fix backward compatibility bug in which earlier version of OL cannot
   get information from higher version if mbatchd, specifically 3.0/3.1
   bqueues cannot get information from 3.3 bqueues.
-- Implement lsb.resources of job slot limit for each user/project.
-- Implement the time window in which resource limits are performed.
-- Fix a broken btop/bbot.
-- Introduce a new rule in config/Makefile.am which creates a file named
   commit in the LSF_ENVDIR, this file contains the git commit number
   of the current build.
-- Fix user group bug in lsb.resources.
-- Fix bresource display invalid max slot/job after mbatchd restart.
-- Refine man5 lsb.resources.5 and man1 bresources.1.
-- Rename example queue/user group/project of lsb.resources.
-- Create 4.0pre1 tag.
-- Implement ABS_RUNLIMIT in lsb.params, if this parameter is enabled
   the runtime specified as bsub -W is not scaled by the execution
   cpu factor.
-- Add bjobs -WF|-WP|-WL for jobs submitted with the -W option.
-- Fix bjobs man page and help for the -W option.
-- Add support of bjobs -p -w.
-- Add “AFFINITY” column in hosts section to specify which hosts support
   cpubinding (affinity) and which ones don’t. OpenLava supports just "y"
   or "n".
-- Fix the config/Makefile.am to create the OpenLava etc directory.
-- Update lsf.conf man page and lsb.hosts man page for cross reference of
   cpu binding.
-- Fix bresources core dump during non-active time window.
-- Fix bresources bug in which current resource usage displays
   duplicated slots/jobs.

3.3.3
======

-- Fix mbatchd bug in which hostnames were sent to the library in a wrong
   causing bjobs to print pending reasons in inverse order.
-- Implement bsub -Jd to support specifying job description to jobs.
-- Implement tech preview of lsb.resources.
-- Implement the bresource command and the lsb_getlimits() API.
-- Fix scheduler bug in which the priority tree was not sorted
   after each job was dispatched.
-- Do not perform bulk slot allocation in the fairshare scheduler
   but distribute slots from higher to low priority jobs.
-- Implement periodic decay of used slots in the fairshare tree.
-- Implement a parameter HIST_MINUTES in lsb.params to control
   the time at which the decay of used slots is performed.
-- Modify the bparams command to display the HIST_MINUTES in its
   long output format.
-- The fairshare layer will log detail information if the logclass
   in lsf.conf is either define as LC_FAIRSHARE or LC_FAIR.
-- Fix the Makefile.am to install bgmod.1 and bjdep.1 man pages.
-- Fix two memory leak in bsub -Jd implementation.
-- Introduce the lsb.resources file.
-- Implement the limit of the number of jobs per project using the
   lsb.resources file.
-- NIOS_RWAIT_SELECT must be set to y or Y in order for nios to wait
   on select to detect that a task has finished.
-- Add template lsb.resources.in.
-- Add man5 lsb.resources.5.
-- Add man1 bresources.1.
-- Implement xdrsize_ResLimitInfoReply() to compute xdr size.
-- Remove typedef of resLimitConf_t, resLimit_t, limitRes_t,
   limitConsumer_t as types are not atomic.
-- Change style declaration, e.g., char* x should be char *x.
-- Replace fname[] with gcc macro __func__.
-- Support specifying job description to job arrays with bsub -Jd.
-- Fix lsb.resources to support Unix user names that are not configured in lsb.users.
-- Implement bresources command to display current resource usage.
-- Implement project account recovery after mbatchd restart.
-- Add BATCH_RESLIMIT_INFO into forkOnRequest() to fork a child for this operation.
-- Fix and XDR decode error in the 3.3 mbatchd if the client has lower versions
   3.0 and 3.1.
-- Fix lsb.resources error when no user is configured in lsb.users.
-- Fix the host group expanding bug in lsb.resources.
-- Fix the resource usage counting bug when enforcing limits.
-- Release 3.3.3 the lsid command returns now 3.3.3 as OpenLava version.
-- Improve bjobs performance ~50 times avoiding sort the SJL by hostname.

3.3.2
======

-- Remove the check if a group name is also a hostname as
   there can be a host on the network not part of the cluster
   with the same name.
-- Add the noheader option for bjobs.
-- Add LSB_MAX_NUM_PROCESSORS, LSB_DJOB_NUMPROC, LSB_JOB_EXECUSER
   and LSB_EFFECTIVE_RSRCREQ variables in the job environment.
-- When a multiprocessor job (-n) specifies memory rusage and span[hosts=1]
   the memory rusage applies to the entire job instead of each slots
   of the job.
-- Fix a bug in lsb.log which was causing 'bsub -K' event being
   badly interpreted by library which printed "Bad event format".
-- Implement built in resource maxslots which represents the MBD
   MXJ parameter. This allows users to sort hosts based on MXJ.
-- Implement NUMA awareness and binding of processes based on the
   machine's NUMA architecture.
-- Fix xdr error in mbatchd if there are no job groups defined
   and user run bjgroup command.
-- Implemented the job group limit 'bgadd -L n /x/y/z'. This allows
   the users to set running job limits at different level of the job group
   hierarchy. If the limit is reached jobs attached to the group will be
   pending with reason "The job group limit has been reached"
-- Renamed the bjgroups command in bjgroup.
-- Implemented bgmod to change the limits of a job group.
-- Fix mbatchd core dump when running egroup.
-- Support bsub with multiple -R options.
-- OpenLava service script sources the custom ulimit scripts.
-- Add preemption for multi-thread jobs when a host is not full.
-- Increase the daemons' port numbers in lsf.conf.in to prevent conflicts
   with other programs.
-- Increase the length of file path to PATH_MAX for bsub -i|-is|-o|-e

3.3.1
======

-- Update the lsb.queues man page describing the OWNERSHIP policy.
-- Improve error message should sbatchd failed to get go ahead
   from mbatchd,
-- Fix a bug in LIM name resolution should the caller be using
   127.0.0.1
-- Add * pattern in job name.
-- Fix bug in parsing all hosts in groups in lsb.hosts.
-- Introduced new performance enhancement parameters in lsb.params.
-- Fix res/nios blocking problems preventing olmake to run
   properly.
-- Fix nios when NIOS_RWAIT_SELECT is set in lsf.conf.

3.3.0 official release
=======

3.3.pre.2
=========
-- Keep the lsb.events and lsb.acct files always open to improve
   file access performance.
-- Implement the switching of the lsb.events asynchronously in mbatchd
   child process.
-- Implemented the bjdep command. bjdep jobid prints the status of
   job dependencies for the given job.
-- Add the bjdep man page.
-- Implement NIOS_RWAIT_SELECT, if configure in lsf.conf then nios
   and ls_rwait() will not use the SIGUSR1 to detect a task that
   has finished. ls_rwait() will select() on a connection with nios
   instead and nios will send a message when a task exits.
-- Fix RES NIOS protocol bug that prevented multiple tasks running on
   the same host from being collected using ls_rwait().
-- Implemented job groups. New commands are bgadd, bgdel and bjgroups.
-- Implemented bjobs -UF unformatted bjobs output.
-- Add LSF_BINDIR in openlava startup scripts.
-- Implement bjobs -g to display only selected job groups.
-- Implement bkill -g to kill only jobs in a specific job group.

3.3.pre.1
=========
-- Fix fairshare algorithm to decrease the number of jobs per user
   only if the job has been started.
-- Fix a bug in ownership policy where wrong runtime comparison was
   performed.
-- Implement the replay of job absolute runtime for ownership policy.
-- Implement floating client feature. If the user configures in lsf.conf
   LIM_ACCEPT_FLOAT_CLIENT=y any host can submit jobs and query the
   daemons without being registered in the lsf.cluster file.

3.2.1
======
-- Fix the parsing algorithm for compact host name.
-- Fix a bug in bmgroup where by mistake the -s and -r options were dropped.
-- Update bmgrop and bugroup man pages.

3.2.0
======
-- Fix main Makefile.am to include the m4 directory.

3.2.beta1
=========
-- Hosts member of a host group can now be specified in a compact format
   for example: (ol[1-3]).
-- Implement configure option --enable-debug. If set the compilation will
   have the -Werror option enabled. By default this option is not set.

3.2.pre
========
-- badmin ckconfig/reconfig displays configuration error and warning
   messages.
-- bsub/bmod/brestart -h displays usage information.
-- Added USER_SHARES information in lsb.users.5
-- Add lsclusters command.
-- Use qsort() to sort jobs in MBD this improves the performance
   of replay and general bsub operations.
-- Introduce a getc2() functions to read lsb.events in memory
   before parsing it.
-- Reduce the number of malloc operations when reading the
   lsb.events file.
-- Implement memory reservation if the queue has SLOT_MEMORY_RESERVE configured
   and the job requests rusage of memory. The jobs has to reserve slots
   on the hosts as well.
-- Implement a correct switching of the lim.events file.
-- Update the bmgroup command to display GROUP_SLOT and MAX_SLOTS
   sections if configured.
-- Update the bqueues command to display slot and memory reservation.
-- Allow batch hosts configured in lsb.hosts to be specified in a
   compact form, i.e. ol[1-3] the mbatchd will expand example this into 3
   hosts ol1, ol2, ol3.
-- Allow OpenLava to use cores as cpus regardless is the machines has
   hyperthreads or not. To enable this behaviour set the variable
   LIM_DEFINE_NCPUS=cores|threads in lsf.conf. Default is threads.
-- Fix bsub bug in which if user specify -n min,max the max was ignored.
-- Make fairshare configuration compatible with the LSF one.

3.1.4
======

-- Fix the problem that if the order of events in lim.events is not correct
   LIM may core dump.
-- Change the ownership of the mbatchd log file to avoid messages going
   to syslog messages file.
-- Prevent LIM from core dumping in findHost() if the input name is NULL.
-- Enable the secondary admin to run 'badmin reconfig'.
-- Fix res nios protocol to pass the correct taskid to nios.

3.1.3
======
-- Corrected the lsf.cluster man page describing LIM communication
   parameters.
-- Enable LIM communication debugging using LC_COMM class and
   LOG_INFO level.
-- Fix bug in announceMaster() which was slowing down announcement
   of slave hosts.
-- Fix a bug in the slave host timer causing slave to send load
   to master more frequently then necessary.
-- Fix counter update in fairshare policy when jobs are suspended
   and resumed.
-- Remove spurious res log messages.

3.1.2
=====
-- Fix a critical mbatchd bug when sorting queues.
-- Enable the openlava.spec file and the rpm.sh script
   which builds the rpm.

3.1.1
=====
-- Fix scheduling bug in which performance could drop if the lowest
   priority queue was always full.
-- Fix the incorrect encoding of bchkpnt command data structure.

3.1.0
======
-- Modify openlava.sh to forcefully terminate sbatchd if SIGTERM does
   not work.
-- Fix LIM bug in which the code was accessing non initialized memory.
-- Fix several memory leaks in LIM.
-- Fix res hang in child handler.
-- Fix memory corruption in mbatchd when dealing with rusage.
-- Fix mbatchd child core dump when processing bugroup command.
-- Improve fairshare performance.
-- Introduce an lsf.conf parameter MBD_MAX_JOB_SCHED which allows site
   to configure how many jobs should MBD attempt to schedule per cycle.
   Typically two times the number of slots.
-- Introduce an lsf.conf parameter MBD_QSORT_JOBS which changes the way
   jobs are inserted and sorted in the PJL. This parameter avoids searching
   the job list upon a new job arrival speeding up the submission and overall
   MBD operations.
-- Fix issues in the job sorting order while using qsort().
-- Increase the array size of LIM clients from 64 to 2014.
-- Fix lim core dump if the lim receives an operation request from
   outside of the OL cluster and it is not LIM_ADD_HOST.
-- Fix bsub < file which was not creating the shell file correctly.
-- Fix rusage bug in which the host could be allocated to a job even with
   not enough resources.
-- Fix memory error in mbatchd when computing job's rusage.
-- Fix various memory leaks

3.1pre1
=======

-- Removed obsolete mls code and infrastructure simplifying user
   management.
-- No longer run mbatchd as root. The uid of the daemon is the one
   of the primary lsb manager.
-- Support memory constraining using cgroup for batch jobs.
-- Developed a built in resource names 'slots' used by mbatchd only.
   This resource allows to pack or stripe jobs on hosts.
-- Developed the rusage or functionality. -R "rusage[x=1||y=2]".
-- Introduced LSB_PROJECT_NAME variable in job and preexec environment.
-- Developed the host group based slot resources. A host group can have
   a resource which represents the number of available slots in that
   group. An ordinary shared resource is configured in the base
   files but also in lsb.hosts in the group section. No elim is needed
   to collect this resource as mbatchd computes the slots automatically.
-- Introduce a mutex to protect critical section in the syslog library.
-- lsaddhost has a new -M option. This indicates the MXJ that should be
   assigned to the batch host, this also sets the number of CPUS available
   on the batch host.
-- Improve scheduling performance when jobs use shared resources.
-- Improve rusage syntax parsing, now spaces between operands are ignored
   and mbatchd does not syntax error.
-- Keep updating man pages.
-- Add the environment variable LSB_EFFECTIVE_RUSAGE. In case a job is using
   the rusage or feature this variable indicates which or clause evaluated
   true dispatching the job.
-- Fix fairshare bug which prevented jobs from being scheduled if groups
   had different shares.
-- Added support for client only hosts. These are host that do not run
   OpenLava daemons but can run commands and submit jobs.
-- Fix mbatchd bug that limited the output and error file to 80 characters
   of length.
-- Enlarged MAXLSFNAMELEN to 1024.
-- Introduce cpu binding using sched_setaffinity()/sched_getaffinity(),
   this feature is enabled by setting SBD_BIND_CPU=y in lsf.conf.
-- Introduced LOAD_UPDATE_INTVL parameter in lsb.params. It controls how
   often MBD calls LIM to update load and shared resources. The default
   value is 15 seconds. Smaller the value and faster is the scheduler
   response to changes in resource values.
-- Fix MBD job lists bug preventing preemption from working correctly.
-- Add -rdynamic option to gcc as new compiler (4.8.3)don't set it by default.
-- Fix the limitation of the select() system call which can handle only 1024
   simultaneous connections. Use poll() instead.
-- Fix potential buffer overrun in lim when running elim.
-- Modify bparams and mbatchd to display maxSbdConnections.
-- Added a new feature MAX_SLOTS in lsb.hosts which is the upper limit
   for the number of usable slots in a hostgroup. The parameter GROUP_SLOTS
   must be enabled in lsb.hosts.
-- Fix wrong calculation of fairshare free slots in the bqueues -l display.

3.0.1
======

-- Fix bug in openlava.in, the variable err was incorrectly defined.
-- Fix xdr version number bug in lsb.xdr.c
-- Fix hostID bug which prevented queue JL/H from working properly.
-- Allow comma as separator together with colon when rusage is
   specified. e.g -R rusage[mem=1,vcs=2]
-- Fix the bug which prevented bsub -K from working.
-- Add script for Intel MPI.
-- Fix mbatchd core dump if the preemption plugin cannot be loaded.
-- Link daemons and commands statically so there is no need to
   set LD_LIBRARY_PATH in the environment.
-- Add LSF_LIBDIR in the lsf.conf.in template.
-- Fix mbatchd core dump when replying partially cleaned job arrays.
-- Modify the resource requirement parser to handle compact syntax,
   like select[swp>1].
-- Implement lim -t to print the architecture, model, cores, sockets
   number of cpus and bogomips of the machine where lim is running.
-- Fix the bug in which sbatchd was logging post exec failure even if
   the post exec ran fine.
-- Reduce or remove sleep times in res.
-- Implement sorting by MXJ in mbatchd.
-- Implement sorting by cpu factor in mbatchd.
-- Fix bjobs -pl core dump.
-- Fix bug in job lists management which leads to mbatchd core dumps.

3.0
====

Major release which includes hierarchical slot fairshare and preemption.

3.0beta
=======
-- Queue level preemption
-- Queue level fairshare
-- Fix LIM core dump when lim.events has bad format
-- Ignore PREEMPTABLE keyword in lsb.queues
-- Implement bpost/bread commands using API
   lsb_msgjob()/lsb_getmsgjob().
-- Implement docker integration
-- Rename the lsb_msgjob()/lsb_getmsgjob() to
   lsb_postjobmsg()/lsb_readjobmsg() for compatibility
-- Fix bug in which every user can shutdown LIM.

2.3-0rc
========
-- Fix the problem of host going periodically -ok.
-- Fix timer initialization in LIM.
-- Fix mbatchd to clean HOST_STAT_EXCLUSIVE when reply lsb.events.
-- Fix job dependencies by name. The dependencies by name incorrectly
   included all jobs belong to the user submitting the jobs.
-- The daemons are not using privilege ports anymore even if
   running as root.
-- Reduce the RES_SLEEP_TIME from 30s to 15s to avoid hosts appearing
   as not available in lim.
-- Remove obsolete INTER_DAEMON_AUTH define.
-- Remove the need for privilege port between res and lim.

2.2
====
-- RPM files are no longer distributed, however they can still be built
   from the tarball.
-- When migrant hosts are added or removed, we must increment or
   decrease the numHosts field in the struct clusterNode (myClusterPtr)
   to allow the periodic master announcement to be sent to dynamic
   nodes in the cluster on master lim restart.
-- Don't call select in rd_select_() when the input file descriptor
   is less than zero, as this causes a buffer overflow abort when
   openlava is built with fortify for debian packages.
-- Fix core dump in bjobs. Since 2.1 hostid start from 0.
-- Fixed a typo in the lsf.cluster man page.
-- Print correct pending reason when hostid is 0.
-- Corrected HACKING file.
-- make install no longer installs the configuration file, this avoids
   overwriting production configuration files.
-- Added -fPIC option to configure.ac - needed when compiling the object
   code into a library for use in third party tools.
-- Fixed bsub -I, now the the job data are correctly sent to the
   user terminal and viceversa.
-- Accept `DESTDIR` environment variable, allowing staged installs.
-- Avoid announcing all LIMs at the same time.
-- Install the config.h in include directory.
-- Fixed lshosts -l core dump.
-- Change the behaviour of the SBD SIGCHLD handler. By default, the
   handler would sleep for 1 second for each child reaped, which would
   cause long waits on machines with lots of slots when lots of jobs
   finished at the same time (e.g. lots of short jobs). The default is
   now to not sleep at all.
-- Count cached memory as free memory.
-- Added error exit message for missing ncurses.h
-- Added MANPATH to shell profile scripts.
-- Allow multiple hosttypes and hostmodels in the shared and cluster file.
   LIM now can be configured with any type and model.