WIP OPA ACCM/Data access enforcement (#29)

* migrated source based tracking to a new prop structure. allow for more metadata * Added attribute classification enforcement * updates * moar * Fixed classification prop disappearing on insert of new objects * Removed some obe logs * moar cleanup * Fix non document types breaking kafka event broadcasting * Scaffolded structure * Fixes, switched over permission eval to hardcoded structure. Need to integrate with OPA * Expanded action authorization support * Initial set of changes * Wired returned opa policies to object level permission evaluators * Added missing root opa url * Implemented delete permission checking * Fixes and other various changes * Bug fixes * Fixed issues with auth - Validation is failing when providing general section for classification * Serialization update * Fixed most bugs * Fixed couple more issues - When creating edges between vertecies there are updates on vertecies which needs to be handled. The action does not come across as create but rather update - * Minor changes - Added todo and removed extra lines * Updated Readme to reflect changes --------- Co-authored-by: Patrick Stevens <[email protected]>
raft-tech · Jun 7, 2024 · fbbb12a · fbbb12a
1 parent 6078472
commit fbbb12a
Show file tree

Hide file tree

Showing 30 changed files with 1,229 additions and 459 deletions.
diff --git a/Raft-README.md b/Raft-README.md
diff --git a/data access enforcement structure.yml b/data access enforcement structure.yml
@@ -0,0 +1,123 @@
+Title: Dynamic Data Driven Access Control
+
+Summary: 
+Raft has requirements to limit data access to users based on their roles, and need to know (NTK), represented by several metadata attributes tied to the user.
+
+To achieve this, Raft is interested in appending query predicates and limiting returned columns on user queries that limit the data accessed to just that which the user is allowed to see. 
+
+IE: 
+SELECT * FROM People -> 
+
+SELECT firstName, lastName, dob FROM People WHERE classification in ["U", "S"] AND releasableTo CONTAINS ["USA"]
+
+The second query has appended predicates that filter out data based on predefined metadata on the record. Additionally the final query on behalf of the user limits the columns that are returned.
+
+We will be use OPA, an open source authorization rules engine, to generate data access restrictions for a user. We will present the data access restrictions for the user in a json form, that can be consumed by the data store to add the current query context and filter data as needed.
+
+The query predicates are defined in a tree structure where the root node is an AND node and the children nodes can be either AND or OR nodes. The leaf nodes are the actual rules that are used to enforce access control. The rules are defined using the `Argument` structure.
+
+
+The below structure is used to define the data access enforcement rules for a given query. The rules are used to enforce access control on the data source. The rules are defined using a tree structure where the root node is an AND node and the children nodes can be either AND or OR nodes. The leaf nodes are the actual rules that are used to enforce access control. The rules are defined using the `Argument` structure.
+
+result
+  create // restrictions on creating new objects
+    - expression
+      ...
+  read // restrictions on reading objects
+    - expression
+    ...
+  update // restrictions on updating objects
+    - expression
+    ...
+  delete // restrictions on deleting objects
+    - expression
+    ...
+
+Below are the json structures we'll use to to define the data access enforcement rules for a given query. 
+
+Base structure. Defines the data access enforcement rules for a given type. The rules are split out by CRUD type. A user can have multiple entries for each CRUD action, to support a real world usecase where a user will have varying access to different combinations of data.
+
+TypeRestriction:
+  id: uuid
+  name: string
+  type: vertex|edge
+  create: Expression[] (nullable)
+  read: Expression[] (nullable)
+  update: Expression[] (nullable)
+  delete: Expression[] (nullable)
+
+// Expression- a nestable structure that can be used to define complex rules. The expression can be a leaf node or a parent node. The parent node can have multiple children nodes. The children nodes can be either AND or OR nodes. The leaf nodes are the actual rules that are used to enforce access control. The rules are defined using the `Argument` structure.
+
+Expression:
+  id: uuid
+  type: AND|OR
+  expression!: Expression[] (nullable)
+  arguments!: Argument[] (nullable)
+
+// A single evaluatable rule that can be used to enforce access control. This can be combined with multiple other arguments to form a complex rule. The argument is defined using the field, operator and value.
+
+Argument:
+  field: string
+  operator: see list below
+  value: string|number|array|boolean
+
+Argument operator options:
+   Value comparison:
+   EQ
+   ANY_OF (target field is any of the provided values)
+
+   Numeric:
+   GT
+   GT_EQ
+   LT
+   LT_EQ
+
+   Array operators (target field is an array)
+   ANY_IN (any of the provided values is present in the target field)
+   ALL_IN (all of the provided values are present in the target field)
+   NONE_IN (none of the provided values are present in the target field)
+
+
+Example Expression:
+
+1. WHERE classification in ["U", "S"] AND releasableTo CONTAINS ["USA"]
+
+  id: 12345-asdf-1342453456
+  type: AND
+  arguments
+  - field: classification
+    operator: ANY_OF
+    value: ["U", "S"]
+  - field: releaseableTo
+    operator: ANY_IN
+    value: ["USA"]
+
+
+2. WHERE classification in ["U", "S"] AND (releasableTo CONTAINS ["USA"] OR noforn = true)
+
+  id: 12345-asdf-1342453456
+  type: AND
+  expression:
+  - id: 13245-43jkafasdf-23423fas
+    type: OR
+    arguments:
+    - field: releaseableTo
+      operator: ANY_IN
+      value: ["USA"]
+    - field: noforn
+      operator: EQ
+      value: true
+  arguments:
+  - field: classification
+    operator: ANY_OF
+    value: ["U", "S"]
+
+
+
+
+
+
+
+
+
+
diff --git a/engine/src/main/java/com/arcadedb/GlobalConfiguration.java b/engine/src/main/java/com/arcadedb/GlobalConfiguration.java
@@ -184,6 +184,7 @@ public Object call(final Object value) {
   KEYCLOAK_CLIENT_ID("arcadedb.keycloakClientId", SCOPE.SERVER, "Keycloak client ID", String.class, "df-backend"),
   KEYCLOAK_REALM("arcadedb.keycloakRealm", SCOPE.SERVER, "Keycloak realm", String.class, "data-fabric"),
   KEYCLOAK_USER_CACHE_EXPIRE("arcadedb.keycloakUserCacheExpire", SCOPE.SERVER, "User cache expire ttl in ms", Long.class, 60000),
+  OPA_ROOT_URL("arcadedb.opaRootUrl", SCOPE.SERVER, "Opa root url", String.class, "http://df-opa:8181"),
 
   DATE_TIME_FORMAT("arcadedb.dateTimeFormat", SCOPE.DATABASE, "Default date time format using Java SimpleDateFormat syntax", String.class,
       "yyyy-MM-dd HH:mm:ss"),
@@ -426,7 +427,7 @@ public Object call(final Object value) {
       "TCP/IP port number used for incoming connections for Mongo plugin. Default is 27017", Integer.class, 27017),
 
   MONGO_HOST("arcadedb.mongo.host", SCOPE.SERVER,
-      "TCP/IP host name used for incoming connections for Mongo plugin. Default is '0.0.0.0'", String.class, "0.0.0.0"),
+      "TCP/IP host name used for incoming connections for Mongo plugin. Default is '0.0.0.0'", String.class, "0.0.0.0"), 
   ;
 
   /**

diff --git a/engine/src/main/java/com/arcadedb/database/DocumentValidator.java b/engine/src/main/java/com/arcadedb/database/DocumentValidator.java
@@ -18,6 +18,7 @@
  */
 package com.arcadedb.database;
 
+import com.arcadedb.database.EmbeddedDatabase.RecordAction;
 import com.arcadedb.exception.ValidationException;
 import com.arcadedb.schema.DocumentType;
 import com.arcadedb.schema.Property;
@@ -59,7 +60,7 @@ public static void verifyDocumentClassificationValidForDeployment(String toCheck
   }
 
   public static void validateClassificationMarkings(final MutableDocument document, 
-          SecurityDatabaseUser securityDatabaseUser) {
+          SecurityDatabaseUser securityDatabaseUser, RecordAction action) {
 
     if (document == null) {
       throw new ValidationException("Document is null");
@@ -79,10 +80,9 @@ public static void validateClassificationMarkings(final MutableDocument document
     }
 
     boolean validSources = false;
-
     // validate sources, if present
     if (document.has(MutableDocument.SOURCES_ARRAY_ATTRIBUTE) && !document.toJSON().getJSONArray(MutableDocument.SOURCES_ARRAY_ATTRIBUTE).isEmpty()) {
-      validateSources(document, securityDatabaseUser);
+      validateSources(document, securityDatabaseUser, action);
       validSources = true;
     }
 
@@ -103,7 +103,7 @@ public static void validateClassificationMarkings(final MutableDocument document
       }
 
       // Validate the user can set the classification of the document. Can't create higher than what you can access.
-      if (!AuthorizationUtils.checkPermissionsOnDocumentToWrite(document, securityDatabaseUser)) {
+      if (!AuthorizationUtils.checkPermissionsOnDocument(document, securityDatabaseUser, action)) {
         throw new ValidationException("User cannot set classification markings on documents higher than or outside their current access.");
       }
 
@@ -125,13 +125,13 @@ public static void validateClassificationMarkings(final MutableDocument document
         throw new ValidationException("Missing classification attributes on document");
       }
 
-      validateAttributeClassificationTagging(document, classificationObj.getJSONObject(MutableDocument.CLASSIFICATION_ATTRIBUTES_PROPERTY), securityDatabaseUser);
+      validateAttributeClassificationTagging(document, classificationObj.getJSONObject(MutableDocument.CLASSIFICATION_ATTRIBUTES_PROPERTY), securityDatabaseUser, action);
     } else if (!validSources){
       throw new ValidationException("Missing overall classification data on document");
     }
   }
 
-  private static void validateAttributeClassificationTagging(final MutableDocument document, final JSONObject attributes, SecurityDatabaseUser securityDatabaseUser) {
+  private static void validateAttributeClassificationTagging(final MutableDocument document, final JSONObject attributes, SecurityDatabaseUser securityDatabaseUser, RecordAction action) {
 
     // confirm each json key in document has a matching key in attributes
     // have counter for each key in document, and decrement when found in attributes
@@ -157,7 +157,7 @@ private static void validateAttributeClassificationTagging(final MutableDocument
       var value = entry.getValue().toString();
 
       if (value != null && value.trim() != "") {
-        if (!AuthorizationUtils.checkPermissionsOnClassificationMarking(value, securityDatabaseUser)){
+        if (!AuthorizationUtils.checkPermissionsOnDocument(document, securityDatabaseUser, action)) {
           throw new ValidationException("User cannot set attribute classification markings on documents higher than or outside their current access.");
         }
       } else {
@@ -175,19 +175,19 @@ private static void validateAttributeClassificationTagging(final MutableDocument
    * Sources are stored in the document as a JSON object, with the key being a numbered list, and the values being the portion marked source id.
    * @param document
    */
-  private static void validateSources(final MutableDocument document, SecurityDatabaseUser securityDatabaseUser) {
+  private static void validateSources(final MutableDocument document, SecurityDatabaseUser securityDatabaseUser, RecordAction action) {
     var sources = document.toJSON().getJSONArray(MutableDocument.SOURCES_ARRAY_ATTRIBUTE);
     sources.forEach(obj -> {
 
       var jo = (JSONObject) obj;
 
-      if (!jo.has(MutableDocument.CLASSIFICATION_PROPERTY)) {
+      if (!jo.has(MutableDocument.CLASSIFICATION_GENERAL_PROPERTY)) {
         throw new ValidationException("Source " + jo + " is missing classification property");
       }
 
-      var classification = jo.getString(MutableDocument.CLASSIFICATION_PROPERTY);
+      var classification = jo.getString(MutableDocument.CLASSIFICATION_GENERAL_PROPERTY);
 
-      if (!AuthorizationUtils.checkPermissionsOnDocumentToWrite(document, securityDatabaseUser)) {
+      if (!AuthorizationUtils.checkPermissionsOnDocument(document, securityDatabaseUser, action)) {
         throw new ValidationException("User cannot set classification markings on documents higher than or outside their current access.");
       }
 
@@ -202,7 +202,7 @@ private static void validateSources(final MutableDocument document, SecurityData
         throw new ValidationException("Invalid classification for source: " + classification);
       }
 
-      validateAttributeClassificationTagging(document, jo.getJSONObject(MutableDocument.CLASSIFICATION_ATTRIBUTES_PROPERTY), securityDatabaseUser);
+      validateAttributeClassificationTagging(document, jo.getJSONObject(MutableDocument.CLASSIFICATION_ATTRIBUTES_PROPERTY), securityDatabaseUser, action);
     });
   }
 

diff --git a/engine/src/main/java/com/arcadedb/database/EmbeddedDatabase.java b/engine/src/main/java/com/arcadedb/database/EmbeddedDatabase.java
@@ -37,11 +37,7 @@
 import com.arcadedb.exception.InvalidDatabaseInstanceException;
 import com.arcadedb.exception.NeedRetryException;
 import com.arcadedb.exception.TransactionException;
-import com.arcadedb.graph.Edge;
-import com.arcadedb.graph.GraphEngine;
-import com.arcadedb.graph.MutableVertex;
-import com.arcadedb.graph.Vertex;
-import com.arcadedb.graph.VertexInternal;
+import com.arcadedb.graph.*;
 import com.arcadedb.index.IndexCursor;
 import com.arcadedb.index.IndexInternal;
 import com.arcadedb.index.TypeIndex;
@@ -60,6 +56,7 @@
 import com.arcadedb.schema.Property;
 import com.arcadedb.schema.Schema;
 import com.arcadedb.schema.VertexType;
+import com.arcadedb.security.AuthorizationUtils;
 import com.arcadedb.security.SecurityDatabaseUser;
 import com.arcadedb.security.SecurityManager;
 import com.arcadedb.serializer.BinarySerializer;
@@ -123,6 +120,10 @@ public class EmbeddedDatabase extends RWLockContext implements DatabaseInternal
   private final        ConcurrentHashMap<String, QueryEngine>    reusableQueryEngines                 = new ConcurrentHashMap<>();
   private              TRANSACTION_ISOLATION_LEVEL               transactionIsolationLevel            = TRANSACTION_ISOLATION_LEVEL.READ_COMMITTED;
 
+  public static enum RecordAction {
+    CREATE, READ, UPDATE, DELETE
+  }
+
   protected EmbeddedDatabase(final String path, final PaginatedComponentFile.MODE mode, final ContextConfiguration configuration, final SecurityManager security,
                              final Map<CALLBACK_EVENT, List<Callable<Void>>> callbacks) {
 
@@ -803,7 +804,7 @@ public void createRecordNoLock(final Record record, final String bucketName, fin
     setDefaultValues(record);
 
     if (record instanceof MutableDocument) {
-      ((MutableDocument) record).validateAndAccmCheck(getContext().getCurrentUser());
+      ((MutableDocument) record).validateAndAccmCheck(getContext().getCurrentUser(), RecordAction.CREATE);
 
       ((MutableDocument) record).set(Utils.CREATED_BY, getCurrentUserName());
       ((MutableDocument) record).set(Utils.CREATED_DATE, LocalDateTime.now());
@@ -877,7 +878,7 @@ public void updateRecord(final Record record) {
       var rec = (MutableDocument) record;
       var context = DatabaseContext.INSTANCE.getContext(rec.database.getDatabasePath());
       if (context.getCurrentUser() != null)
-        rec.validateAndAccmCheck(context.getCurrentUser());
+        rec.validateAndAccmCheck(context.getCurrentUser(), RecordAction.UPDATE);
     }
 
     // INVOKE EVENT CALLBACKS
@@ -992,7 +993,15 @@ public void updateRecordNoLock(final Record record, final boolean discardRecordA
   @Override
   public void deleteRecord(final Record record) {
     executeInReadLock(() -> {
+      if (record instanceof MutableDocument) {
+        var rec = (MutableDocument) record;
+        var context = DatabaseContext.INSTANCE.getContext(rec.database.getDatabasePath());
+        if (context.getCurrentUser() != null)
+            rec.validateAndAccmCheck(context.getCurrentUser(), RecordAction.DELETE);
+      }
+
       deleteRecordNoLock(record);
+
       return null;
     });
   }

diff --git a/engine/src/main/java/com/arcadedb/database/MutableDocument.java b/engine/src/main/java/com/arcadedb/database/MutableDocument.java
@@ -18,6 +18,7 @@
  */
 package com.arcadedb.database;
 
+import com.arcadedb.database.EmbeddedDatabase.RecordAction;
 import com.arcadedb.exception.ValidationException;
 import com.arcadedb.schema.DocumentType;
 import com.arcadedb.schema.Property;
@@ -42,7 +43,8 @@ public class MutableDocument extends BaseDocument implements RecordInternal {
   protected boolean dirty = false;
 
   public static final String CLASSIFICATION_PROPERTY = "classification";
-  public static final String CLASSIFICATION_GENERAL_PROPERTY = "general";
+  public static final String CLASSIFICATION_GENERAL_PROPERTY = "clearance";
+  public static final String CLASSIFICATION_RELEASABLE_TO = "releasableTo";
   public static final String CLASSIFICATION_ATTRIBUTES_PROPERTY = "attributes";
   public static final String CLASSIFICATION_MARKED = "classificationMarked";
 
@@ -171,9 +173,9 @@ public List<Property> getAccmProperties() {
   /**
    * Triggers the native required property valiation of arcade, as well as the one time ACCM validation.
    * ACCM validation follows a different recursive type checking pattern than arcade, so it is done separately.
- * @param securityDatabaseUser
+   * @param securityDatabaseUser
    */
-  public void validateAndAccmCheck(SecurityDatabaseUser securityDatabaseUser) {
+  public void validateAndAccmCheck(SecurityDatabaseUser securityDatabaseUser, RecordAction action) {
 
     /**
      * Skip validitng during initial edge creation, as they don't have properties set yet.
@@ -193,9 +195,8 @@ public void validateAndAccmCheck(SecurityDatabaseUser securityDatabaseUser) {
      */ 
     try {
       if (this.getDatabase().getSchema().getEmbedded().isClassificationValidationEnabled()) {
-
         if (toJSON() != null) {
-          DocumentValidator.validateClassificationMarkings(this, securityDatabaseUser);
+          DocumentValidator.validateClassificationMarkings(this, securityDatabaseUser, action);
           set(CLASSIFICATION_MARKED, true);
         }
       }

diff --git a/engine/src/main/java/com/arcadedb/engine/BucketIterator.java b/engine/src/main/java/com/arcadedb/engine/BucketIterator.java
@@ -18,10 +18,7 @@
  */
 package com.arcadedb.engine;
 
-import com.arcadedb.database.Binary;
-import com.arcadedb.database.Database;
-import com.arcadedb.database.DatabaseInternal;
-import com.arcadedb.database.RID;
+import com.arcadedb.database.*;
 import com.arcadedb.database.Record;
 import com.arcadedb.exception.DatabaseOperationException;
 import com.arcadedb.log.LogManager;