Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a function to get the size of a collection/table in bytes. #272

Merged
merged 39 commits into from
Aug 10, 2023
Merged
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
2711213
fixed stuff
faisalill May 26, 2023
e645817
fixed indentation
faisalill May 29, 2023
32166d1
indentation cleanup
faisalill May 29, 2023
de08dca
fixed indentation
faisalill May 30, 2023
aec995d
fixed indentation for all adapters
faisalill May 30, 2023
c7b797f
Merge branch 'utopia-php:main' into main
faisalill Jun 1, 2023
d66d2f8
fixed mysql test issue
faisalill Jun 3, 2023
6afaf6f
getSizeOfCollection analyzes the table before getting the size
faisalill Jun 9, 2023
540fe3c
added analyzeCollection function
faisalill Jun 12, 2023
a305282
Merge branch 'utopia-php:main' into main
faisalill Jun 12, 2023
a664c6b
called analyzeCollection from getSizeOfCollection
faisalill Jun 12, 2023
7b0a74a
removed Return from comments
faisalill Jun 12, 2023
7419c99
Merge branch 'utopia-php:main' into main
faisalill Jun 18, 2023
2e60d71
remove analyzeCollection and changed query for mariadb and mysql
faisalill Jun 21, 2023
416fa5f
cleaner code
faisalill Jun 21, 2023
b6c4a03
added camel case for index
faisalill Jun 21, 2023
682ca9f
removed analyze for sqlite
faisalill Jun 21, 2023
3dc00b3
cleaner code
faisalill Jun 21, 2023
ecda582
changed mariadb query
faisalill Jun 22, 2023
4a6ce3f
changed query of getSizeOfCollection
faisalill Jun 29, 2023
faab827
fixed lint issues
faisalill Jul 12, 2023
51ffb2c
fixed phpstan issue
faisalill Jul 13, 2023
2761977
removed mytest from composer.json
faisalill Jul 13, 2023
4c13af1
Merge branch 'utopia-php:main' into main
faisalill Jul 18, 2023
af69df4
removed mytest from composer.json
faisalill Jul 18, 2023
ac21574
Add explanation for asserting a constant with size difference
faisalill Jul 18, 2023
7abfde5
Fix lint issue
faisalill Jul 18, 2023
21550ff
Merge branch 'utopia-php:main' into main
faisalill Jul 22, 2023
d12f27b
Made changes as per review
faisalill Jul 26, 2023
9e96dc0
Merge branch 'main' of https://github.com/faisalill/database
faisalill Jul 26, 2023
a0ec416
Apply suggestions from code review
abnegate Jul 26, 2023
ff872db
Make changes as per review
faisalill Jul 27, 2023
06fe605
Add full text test
faisalill Aug 8, 2023
1763e86
Change MongoDB return value
faisalill Aug 10, 2023
1ebd9f2
Remove mytest script from composer.json
faisalill Aug 10, 2023
8484dc8
Use better collection name
faisalill Aug 10, 2023
5048976
Apply suggestions from code review
abnegate Aug 10, 2023
87adb16
Merge branch 'main' into main
abnegate Aug 10, 2023
3b65416
Fix lint
abnegate Aug 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,4 +244,4 @@ Navigate to `localhost:8708` to visualize query results.

## Copyright and license

The MIT License (MIT) [http://www.opensource.org/licenses/mit-license.php](http://www.opensource.org/licenses/mit-license.php)
The MIT License (MIT) [http://www.opensource.org/licenses/mit-license.php](http://www.opensource.org/licenses/mit-license.php)
9 changes: 9 additions & 0 deletions src/Database/Adapter.php
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,15 @@ abstract public function sum(string $collection, string $attribute, array $queri
*/
abstract public function count(string $collection, array $queries = [], ?int $max = null): int;

/**
* Get Collection Size
*
* @param string $collection
* @return int
* @throws DatabaseException
*/
abstract public function getSizeOfCollection(string $collection): int;

/**
* Get max STRING limit
*
Expand Down
40 changes: 40 additions & 0 deletions src/Database/Adapter/MariaDB.php
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,46 @@ public function createCollection(string $name, array $attributes = [], array $in
return true;
}

/**
* Get Collection Size
* @param string $collection
* @return int
* @throws DatabaseException
*/
public function getSizeOfCollection(string $collection): int
{
$collection = $this->filter($collection);
$collection = $this->getNamespace() . '_' . $collection;
$database = $this->getDefaultDatabase();
$name = $database . '/' . $collection;
$permissions = $database . '/' . $collection . '_perms';

faisalill marked this conversation as resolved.
Show resolved Hide resolved
$collectionSize = $this->getPDO()->prepare("
SELECT SUM(FS_BLOCK_SIZE + ALLOCATED_SIZE)
FROM INFORMATION_SCHEMA.INNODB_SYS_TABLESPACES
WHERE NAME = :name
");

$permissionsSize = $this->getPDO()->prepare("
SELECT SUM(FS_BLOCK_SIZE + ALLOCATED_SIZE)
FROM INFORMATION_SCHEMA.INNODB_SYS_TABLESPACES
WHERE NAME = :permissions
faisalill marked this conversation as resolved.
Show resolved Hide resolved
abnegate marked this conversation as resolved.
Show resolved Hide resolved
faisalill marked this conversation as resolved.
Show resolved Hide resolved
");
abnegate marked this conversation as resolved.
Show resolved Hide resolved

$collectionSize->bindParam(':name', $name);
$permissionsSize->bindParam(':permissions', $permissions);

try {
$collectionSize->execute();
$permissionsSize->execute();
$size = $collectionSize->fetchColumn() + $permissionsSize->fetchColumn();
} catch (PDOException $e) {
throw new DatabaseException('Failed to get collection size: ' . $e->getMessage());
}

return $size;
}

faisalill marked this conversation as resolved.
Show resolved Hide resolved
/**
* Delete Collection
* @param string $id
Expand Down
29 changes: 29 additions & 0 deletions src/Database/Adapter/Mongo.php
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,35 @@ public function listCollections(): array
return $list;
}

/**
* Get Collection Size
* @param string $collection
* @return int
* @throws DatabaseException
*/
public function getSizeOfCollection(string $collection): int
{
$namespace = $this->getNamespace();
$collection = $this->filter($collection);
$collection = $namespace. '_' . $collection;

$command = [
'collStats' => $collection,
'scale' => 1
];

try {
$result = $this->getClient()->query($command);
if (is_object($result) && property_exists($result, 'size')) {
return $result->size;
} else {
throw new DatabaseException('Failed to get collection size');
faisalill marked this conversation as resolved.
Show resolved Hide resolved
}
faisalill marked this conversation as resolved.
Show resolved Hide resolved
} catch(Exception $e) {
throw new DatabaseException('Failed to get collection size: ' . $e->getMessage());
}
}

faisalill marked this conversation as resolved.
Show resolved Hide resolved
/**
* Delete Collection
*
Expand Down
41 changes: 41 additions & 0 deletions src/Database/Adapter/MySQL.php
Original file line number Diff line number Diff line change
Expand Up @@ -78,4 +78,45 @@ protected function processException(PDOException $e): void

throw $e;
}

/**
* Get Collection Size
* @param string $collection
* @return int
* @throws DatabaseException
*/
public function getSizeOfCollection(string $collection): int
{
$collection = $this->filter($collection);
$collection = $this->getNamespace() . '_' . $collection;
$database = $this->getDefaultDatabase();
$name = $database . '/' . $collection;
$permissions = $database . '/' . $collection . '_perms';

$collectionSize = $this->getPDO()->prepare("
SELECT SUM(FS_BLOCK_SIZE + ALLOCATED_SIZE)
FROM INFORMATION_SCHEMA.INNODB_TABLESPACES
WHERE NAME = :name
");
faisalill marked this conversation as resolved.
Show resolved Hide resolved

$permissionsSize = $this->getPDO()->prepare("
SELECT SUM(FS_BLOCK_SIZE + ALLOCATED_SIZE)
FROM INFORMATION_SCHEMA.INNODB_TABLESPACES
WHERE NAME = :permissions
");
faisalill marked this conversation as resolved.
Show resolved Hide resolved
faisalill marked this conversation as resolved.
Show resolved Hide resolved

abnegate marked this conversation as resolved.
Show resolved Hide resolved

$collectionSize->bindParam(':name', $name);
$permissionsSize->bindParam(':permissions', $permissions);

try {
$collectionSize->execute();
$permissionsSize->execute();
$size = $collectionSize->fetchColumn() + $permissionsSize->fetchColumn();
} catch (PDOException $e) {
throw new DatabaseException('Failed to get collection size: ' . $e->getMessage());
}

return $size;
}
}
35 changes: 35 additions & 0 deletions src/Database/Adapter/Postgres.php
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,41 @@ public function createCollection(string $name, array $attributes = [], array $in
return true;
}

/**
* Get Collection Size
* @param string $collection
* @return int
* @throws DatabaseException
*
*/
public function getSizeOfCollection(string $collection): int
{
$collection = $this->filter($collection);
$name = $this->getSQLTable($collection);
$permissions = $this->getSQLTable($collection . '_perms');

$collectionSize = $this->getPDO()->prepare("
SELECT pg_total_relation_size(:name);
");

$permissionsSize = $this->getPDO()->prepare("
SELECT pg_total_relation_size(:permissions);
");

$collectionSize->bindParam(':name', $name);
$permissionsSize->bindParam(':permissions', $permissions);

try {
$collectionSize->execute();
$permissionsSize->execute();
$size = $collectionSize->fetchColumn() + $permissionsSize->fetchColumn();
} catch (PDOException $e) {
throw new DatabaseException('Failed to get collection size: ' . $e->getMessage());
}

return $size;
}

faisalill marked this conversation as resolved.
Show resolved Hide resolved
/**
* Delete Collection
*
Expand Down
36 changes: 36 additions & 0 deletions src/Database/Adapter/SQLite.php
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,42 @@ public function createCollection(string $name, array $attributes = [], array $in
return true;
}

/**
* Get Collection Size
* @param string $collection
* @return int
* @throws DatabaseException
*
*/
public function getSizeOfCollection(string $collection): int
{
$collection = $this->filter($collection);
$namespace = $this->getNamespace();
$name = $namespace . '_' . $collection;
$permissions = $namespace . '_' . $collection . '_perms';

$collectionSize = $this->getPDO()->prepare("
SELECT SUM(\"pgsize\") FROM \"dbstat\" WHERE name=:name;
");

$permissionsSize = $this->getPDO()->prepare("
SELECT SUM(\"pgsize\") FROM \"dbstat\" WHERE name=:name;
");

$collectionSize->bindParam(':name', $name);
$permissionsSize->bindParam(':name', $permissions);

try {
$collectionSize->execute();
$permissionsSize->execute();
$size = $collectionSize->fetchColumn() + $permissionsSize->fetchColumn();
} catch (PDOException $e) {
throw new DatabaseException('Failed to get collection size: ' . $e->getMessage());
}

return $size;
}

/**
* Delete Collection
* @param string $id
Expand Down
12 changes: 12 additions & 0 deletions src/Database/Database.php
Original file line number Diff line number Diff line change
Expand Up @@ -789,6 +789,18 @@ public function listCollections(int $limit = 25, int $offset = 0): array
return $result;
}

/**
* Get Collection Size
*
* @param string $collection
*
* @return int
*/
public function getSizeOfCollection(string $collection): int
{
return $this->adapter->getSizeOfCollection($collection);
}

/**
* Delete Collection
*
Expand Down
33 changes: 33 additions & 0 deletions tests/Database/Base.php
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,39 @@ public function testCreateListExistsDeleteCollection(): void
$this->assertEquals(false, static::getDatabase()->exists($this->testDatabase, 'actors'));
}

public function testSizeCollection(): void
{
static::getDatabase()->createCollection('sizeTest1');
static::getDatabase()->createCollection('sizeTest2');

$size1 = static::getDatabase()->getSizeOfCollection('sizeTest1');
$size2 = static::getDatabase()->getSizeOfCollection('sizeTest2');
$sizeDifference = abs($size1 - $size2);
// Size of an empty collection returns either 172032 or 167936 bytes randomly
// Therefore asserting with a tolerance of 5000 bytes
$byteDifference = 5000;
$this->assertLessThan($byteDifference, $sizeDifference);

static::getDatabase()->createAttribute('sizeTest2', 'string1', Database::VAR_STRING, 128, true);
static::getDatabase()->createAttribute('sizeTest2', 'string2', Database::VAR_STRING, 254 + 1, true);
static::getDatabase()->createAttribute('sizeTest2', 'string3', Database::VAR_STRING, 254 + 1, true);
static::getDatabase()->createIndex('sizeTest2', 'index', Database::INDEX_KEY, ['string1', 'string2', 'string3'], [128, 128, 128]);

$loopCount = 40;

for ($i = 0; $i < $loopCount; $i++) {
static::getDatabase()->createDocument('sizeTest2', new Document([
'string1' => 'string1' . $i,
'string2' => 'string2' . $i,
'string3' => 'string3' . $i,
]));
}

$size2 = static::getDatabase()->getSizeOfCollection('sizeTest2');

$this->assertGreaterThan($size1, $size2);
}

public function testCreateDeleteAttribute(): void
{
static::getDatabase()->createCollection('attributes');
Expand Down