Resource and Issue refactor

[bunjiboys]

This is a major refactor of how resources and issues are managed from a code perspective, similar to the Account Refactor (#173). All the changes in the PR are written to be fully backwards compatible with the existing codebase, albeit some functionality has been marked as deprecated in favor of the newer improved access patterns.

Property access and updating

Instead of manually implementing properties through @property methods on the object classes, you instead simply provide the property metadata and the Base classes will handle the property access.

Old code

class VPC(BaseResource):
    """VPC Object"""
    resource_type = 'aws_vpc'
    resource_name = 'VPC'

    # region Object properties
    @property
    def cidr_v4(self):
        """ Returns the IPv4 CIDR block associated with the VPC
        Returns:
            `str`
        """
        return self.get_property('cidr_v4').value

    @property
    def is_default(self):
        """ Returns whether the VPC is the Default VPC

        Returns:
            `boolean`


        """
        return self.get_property('is_default').value

    @property
    def state(self):
        """ Returns the current state of the VPC (pending/available)
        Returns:
            `str`
        """
        return self.get_property('state').value

    @property
    def vpc_flow_logs_status(self):
        """ Returns the configured state of VPC Flow Logs
        Returns:
            `str`
        """
        return self.get_property('vpc_flow_logs_status').value

    def vpc_flow_logs_log_group(self):
        """ Returns the CloudWatch Log Group associated with the VPC Flow Logs
        Returns:
            `str`
        """
        return self.get_property('vpc_flow_logs_log_stream').value

    # end of region

    def update(self, data, properties):
        """Updates the object information based on live data, if there were any changes made. Any changes will be
        automatically applied to the object, but will not be automatically persisted. You must manually call
        `db.session.add(vpc)` on the object.
        Args:
            data (bunch): Data fetched from AWS API
            properties (bunch): Properties of the VPC and CloudWatch Log Group as fetched from AWS API
        Returns:
            True if there were any changes to the object, else false
        """

        updated = self.set_property('cidr_v4', data.cidr_block)
        updated |= self.set_property('is_default', data.is_default)
        updated |= self.set_property('state', data.state)
        updated |= self.set_property('vpc_flow_logs_status', properties['vpc_flow_logs_status'])
        updated |= self.set_property('vpc_flow_logs_log_stream', properties['vpc_flow_logs_log_group'])

        tags = {x['Key']: x['Value'] for x in data.tags or {}}
        existing_tags = {x.key: x for x in self.tags}

        # Check for new tags
        for key, value in list(tags.items()):
            updated |= self.set_tag(key, value)

        # Check for updated or removed tags
        for key in list(existing_tags.keys()):
            if key not in tags:
                updated |= self.delete_tag(key)

        return updated

becomes

class VPC(BaseResource):
    """VPC Object"""
    resource_type = 'aws_vpc'
    resource_name = 'VPC'
    resource_properties = [
        ResourceProp(key='cidr_v4', name='IPv4 CIDR', type='string', show=True),
        ResourceProp(key='is_default', name='Default VPC', type='bool', show=True),
        ResourceProp(key='state', name='STate', type='string', show=True),
        ResourceProp(key='vpc_flow_logs_status', name='VPC Flow Log Status', type='string', show=True),
        ResourceProp(key='vpc_flow_logs_log_group', name='VPC Flow Log Group', type='string', show=True),
    ]

Since the metadata for the properties of each resource and issue type are new expressed in an easily digestable format, this also allows for generic resource view generation with good naming, which in almost all cases entirely removes the need for manually implementing any frontend code when adding a new type of resource or issue.

The update BaseResource and BaseIssue class added a couple of new methods and properties to better support generic displays, while deprecating some older functionality.

New BaseResource methods

find

This is an updated version of the existing search method, but where the search method would only return objects of the same type as the class it was called on, the find method will allow you to return a collection of multiple resource types in a single query, allowing you to control which types to return by passing in a list of resource types through the resource_types method argument.

In order to provide similar functionality if called on an explicit resource type, it will only return results of that resource type, unless you also pass in one or more resource_types as an argument. Below is an example of first the old search method and then the new find method

Old:

In [3]: EC2Instance.search()
[09:41:14] cloud_inquisitor.utils The `search` method is deprecated, please use the `find` function instead
Out[3]:
(48,
 [<cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106e597b8>,
  <cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106e59898>,
  <cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106e596a0>,
  ......
  <cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106e6a7b8>])

New:

In [4]: EC2Instance.find()
Out[4]:
(48,
 [<cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106e7b668>,
  <cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106e7b3c8>,
  <cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106e7b2b0>,
  <cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106e7b5f8>,
  ......
  <cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106eb60f0>])

However, if you use the BaseResource.find method or pass in a list of resource_types to the find method, it will return a mixed collection instead.

BaseResource

In [5]: BaseResource.find()
Out[5]:
(249,
 [<cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106ebbd68>,
  ......
  <cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106e7aba8>,
  <cloud_inquisitor.plugins.types.resources.AMI at 0x106ec1438>,
  ......
  <cloud_inquisitor.plugins.types.resources.AMI at 0x106eb0e48>,
  <cloud_inquisitor.plugins.types.resources.CloudFrontDist at 0x106eb00f0>,
  <cloud_inquisitor.plugins.types.resources.DNSRecord at 0x106ecec18>,
  ......
  <cloud_inquisitor.plugins.types.resources.DNSRecord at 0x106eb0128>])

In [6]:

Typed search

In [8]: EC2Instance.find(resource_types=['aws_ec2_instance', 'aws_ebs_volume', 'dns_zone', 'dns_record'])
Out[8]:
(155,
 [<cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106ecbe48>,
  ......
  <cloud_inquisitor.plugins.types.resources.EC2Instance at 0x106ecd898>,
  <cloud_inquisitor.plugins.types.resources.DNSRecord at 0x106e36c88>,
  ......
  <cloud_inquisitor.plugins.types.resources.DNSRecord at 0x106efedd8>,
  <cloud_inquisitor.plugins.types.resources.DNSZone at 0x106efe588>,
  <cloud_inquisitor.plugins.types.resources.EBSVolume at 0x106efe278>,
  ......
  <cloud_inquisitor.plugins.types.resources.EBSVolume at 0x106eb9f60>])

All the returned objects will be cast into their proper resource type objects, allowing use of all the standard functionality on the objects as you'd expect.

Zero-indexed

One important difference between search and find is that the new find method is zero-indexed for pagination

update_resource / update_issue

This method handles updating any resource or issue objects properties, regardless of the type. It also allows for partial updates so that you only have to pass just the updated parameters for the object, and not have to send the entire set of properties when only some were changed.

For resources with tags, it does require you to pass the full set of tags in for the resource, as tags are handled as a set resource and not individual properties

Example

instance = EC2Instance.get('i-1234567')
instance.update_resource(
    properties={
        'instance_type': 'm5.large'
    },
    tags={
        'Name': 'MyTestInstance'
    }
)

[TheM0ng00se]

Hey @bunjiboys

Thanks for submitting this, we've some repo consolidation coming and core changes and will need time to review this...it likely won't happen until next year

[TheM0ng00se]

Hey @bunjiboys , we have some significant changes coming to backend and frontend so while a lot of this looks good, we'd probably need you to resubmit after the 2.1 release. Thanks