Skip to content

Home

Jump to bottom
kostas-alexis edited this page Oct 22, 2023 · 16 revisions

ORACLE: Library of Deep Learning-based Safe Navigation Methods

Welcome to the wiki for ORACLE-family of methods!

This wiki will guide you through the installation and running of the package along with documentation of the package.

Method Overview

A family of learning-based methods for (Visually-Attentive) Uncertainty-Aware Navigation is presented in this repo, including ORACLE, A-ORACLE, and seVAE-ORACLE.

The problem considered in this work is that of autonomous uncertainty-aware and visually attentive aerial robot navigation. The method explicitly assumes no access to the map of the environment (neither offline nor online) and no information for the robot position but only a partial state estimate of the robot combined with the real-time depth data and a 2D detection mask representing the interestingness of every region within an angle- and range-constrained sensor frustum. We assume that there is a global planner providing the 3D unit goal vector $\mathbf{n}^g_t$ to the robot (e.g., for exploration or inspection), possibly by having access to a topological map of the environment. Given the above, the focus is on designing a local safe navigation planner to head towards the goal vector and not only avoid obstacles (ORACLE and seVAE-ORACLE) but simultaneously pay attention to interesting areas (A-ORACLE).

ORACLE and A-ORACLE

The below video explains the functionality overview of ORACLE and A-ORACLE:

explanation_slide_multiples-cover_v2.mp4

ORACLE_overview

The algorithmic architecture of Attentive ORACLE (A-ORACLE) and ORACLE: We design two deep neural networks to efficiently estimate the uncertainty-aware collision score and the information gains for multiple action sequences, namely the Collision Prediction Network (CPN) and Information gain Prediction Network (IPN), respectively. Both networks assume access to a) either the depth image (CPN) or the stacked matrix of the current depth image and the detection mask (IPN), alongside b) the estimates of the robot’s linear velocities, $z$-axis angular velocity, and roll/pitch angles and c) candidate action sequences in a Motion Primitives Library (MPL). Notably, CPN utilizes $\mathbf{m}_1$ representing the current mean value of $\mathbf{s}_t$ and $\mathbf{m}_2 ... \mathbf{m}_{N_\Sigma}$ representing the remaining sigma points of the Unscented Transform to account for the uncertainty in the robot's partial state estimate, while an ensemble of CPNs is used to account for the epistemic uncertainty of the neural network model. The predicted uncertainty-aware collision cost $\hat{c}^{uac}$, information gain $\hat{g}$, and a unit goal vector $\mathbf{n}^g_t$ given by a high-level global planner are used to choose the optimal action sequence to be executed in a receding horizon fashion. When the IPN is not engaged, the method reduces to ORACLE method which ensures safe uncertainty-aware map-less navigation.

seVAE-ORACLE

While the above methods can transfer well to the real system, they require a fairly expensive and heuristic pre-processing step on the raw depth images to mitigate the discrepancies between the real and simulated depth images (such as a) missing information, b) loss of detail). Although Deep Ensembles method can (passively) account for the depth image noise by considering them as novel out-of-distribution input, having a pipeline that can incorporate directly noisy real-world exteroceptive sensor input (in addition to simulation data) is beneficial, especially with hard-to-perceive thin obstacles. We address this problem by proposing a modularized learning-based method based on a Semantically-enhanced Variational Autoencoder (seVAE).

The below video explains the functionality overview of seVAE-ORACLE.

vae-oracle-explanation-vF.mp4

seVAE_overview

The algorithmic architecture of seVAE-ORACLE: we propose a modularized approach involving the seVAE and the Collision Prediction Network (CPN). The seVAE encodes the input depth image $\mathbf{x}_t$ into the latent representation $\boldsymbol{\mu}_t$ which is used by the CPN to predict the collision scores $\hat{\mathbf{c}}^{col}_{t+1:t+T+1}$ for each action sequence $\mathbf{a}_{t:t+T}$ in the motion primitives library. Notably, the seVAE is trained with both real-world and simulated depth images to compress the input data, while preserving semantically-labeled thin obstacles and handling invalid pixels in the depth sensor's output. Furthermore, the method utilizes $N_{\Sigma}$ sigma points calculated based on $\mathbf{s}_t$ and $\boldsymbol{\Sigma}_t$ and an ensemble of CPNs to calculate the uncertainty-aware collision score $\hat{c}^{uac}$.

Acknowledgements

This open-source release is based upon work supported by a) the Research Council of Norway project SENTIENT (Project No. 321435), b) the Air Force Office of Scientific Research under award number FA8655-21-1-7033, and c) the Horizon Europe project DIGIFOREST (EC 101070405).

Contents

For specific instructions please visit the respective pages:

  1. Installation
  2. Demo Simulation
  3. Experimental results
  4. Parameters
  5. Planner interface and planner's behavior tree
  6. Training navigation policy for your own drone
Clone this wiki locally