5.1.0 beta & 4.8.0 (#615)

.gitignore

@@ -5,3 +5,4 @@ ignore/*
 *.o
 *.so
 sql/*.sql
+test/not_working_yet/*

CHANGELOG.md

@@ -1,3 +1,38 @@
+5.1.0
+=====
+NEW FEATURES
+------------
+ - Support LIST partitioning for single value integers. (Issue #236)
+ - Add explicit ordering of partition set maintenance. (Issue #497)
+    - A new column has been added to the `part_config` table: `maintenance_order`. If set, the partition sets will run in increasing numerical order
+    - Defaults to NULL and has no requirement
+    - NULL values will always come after numbered sets in no guaranteed order
+ - Added new column `part_config.maintenance_last_run` to track the last datetime that maintenance was run for that partition set. Timestamp is only updated if maintenance for that partition set completed successfully so can be used as a monitoring metric.
+ - REPLICA IDENTITY is now automatically inherited from the parent table to all children (#502)
+    - Note for existing partition sets, this will only apply to newly created child tables. Existing child tables will need to be manually updated.
+ - EXPERIMENTAL - Support numeric partitioning (Issue #265)
+    - Note that while the partition column may now be of type numeric, the partitioning interval must still be a whole integer value
+    - Please evaluate this feature carefully before using in production and feel free to open issues or discussions on the Github repository for both positive and negative feedback. Positive feedback will speed up moving this feature out of experimental.
+
+BUGFIXES
+--------
+ - Remove child tables from publication during retention that keeps tables (Github Issue #523)
+ - Allow partition maintenance to be called on replicas without error. Calling maintenance on a replica will do nothing and exit cleanly. Allows for setting up consistent cronjobs between failover systems. (Github Issue #569)
+ - Properly inherit tablespaces (Github Issue #609)
+    - This was a regression in 5.0 that mistakenly stopped working. Tablespace inheritance still works as expected in 4.x.
+ - Allow `infinite_time_partitions` flag to work even if the partition set has no data. This can happen in partition sets with retention and low data writes (Github Issue #585)
+ - Fixed edge case where partition sets with zero data would still get new partitions created. Triggered by partman maintaining multiple partition sets and maintenance running on one that had data followed by one that did not.
+ - Fixed `partition_data` functions throwing an error if the source table was not in the same schema as the parent table. (Github Issue #639)
+
+
+4.8.0
+=====
+BUG FIXES
+---------
+ - Added `pg_analyze` parameter to `partition_gap_fill` function to allow skipping the analyze of a partition set if the gap fill actually creates new partitions. Note this is not an option in 5.x since the analyze step was refactored and never runs automatically during a call to the gap fill function anymore.
+ - Note that there is no way to directly install version 4.8.0. A prior version must be installed first and then upgraded to 4.8.0. It is recommended that you migrate to the latest major version for continued support of pg_partman.
+
+
 5.0.0 & 5.0.1
 =============
 UPGRADE NOTES

LICENSE.txt

@@ -4,7 +4,7 @@ LICENSE AND COPYRIGHT
 PG Partition Manager (pg_partman) is released under the PostgreSQL License, a liberal Open Source license, similar to the BSD or MIT licenses.
 
 Copyright (c) 2017 OmniTI, Inc.
-Copyright (c) 2018-2023 Crunchy Data Solutions, Inc.
+Copyright (c) 2018-2024 Crunchy Data Solutions, Inc.
 
 Permission to use, copy, modify, and distribute this software and its documentation for any purpose, without fee, and without a written agreement is hereby granted, provided that the above copyright notice and this paragraph and the following two paragraphs appear in all copies.
 

META.json

@@ -1,7 +1,7 @@
 {
     "name": "pg_partman",
     "abstract": "Extension to manage partitioned tables by time or ID",
-    "version": "5.0.1",
+    "version": "5.1.0",
     "maintainer": [
         "Keith Fiske <[email protected]>"
     ],
@@ -20,9 +20,9 @@
     },
     "provides": {
         "pg_partman": {
-            "file": "sql/pg_partman--5.0.1.sql",
+            "file": "sql/pg_partman--5.1.0.sql",
             "docfile": "doc/pg_partman.md",
-            "version": "5.0.1",
+            "version": "5.1.0",
             "abstract": "Extension to manage partitioned tables by time or ID"
         }
     },

doc/pg_partman.md

@@ -5,7 +5,9 @@ About
 -----
 PostgreSQL Partition Manager is an extension to help make managing time or number/id based table partitioning easier. It has many options, but usually only a few are needed, so it's much easier to use than it may first appear (and definitely easier than implementing it yourself).
 
-As of version 5.0.1, the minimum version of PostgreSQL required is 14 and trigger-based partitioning is no longer supported. All partitioning is done using built-in declarative partitioning. Currently only ranged partitioning is supported for time- and number-based intervals. Version 4.x of pg_partman, which still has trigger-based support, is no longer in active development and will only be receiving critical bug fixes for a limited time. If partitioning is a critical part of your infrastructure, please make plans to upgrade in the near future.
+As of version 5.0.1, the minimum version of PostgreSQL required is 14 and trigger-based partitioning is no longer supported. All partitioning is done using built-in declarative partitioning. Ranged partitioning is supported for time- and number-based intervals. List partitioning is supported for number-based partitioning when the interval is 1. As of version 5.1, EXPERIMENTAL support of numeric/decimal values is available for number-based partitioning, but the interval must still be an integer. Feedback on numeric partitioning, good or bad, is appreciated to accelerate it out of experimental.
+
+Version 4.x of pg_partman, which still has trigger-based support, is no longer in active development and will only be receiving critical bug fixes until 14 is the last supported version of PostgreSQL. If partitioning is a crucial part of your infrastructure, please make plans to upgrade in the very near future.
 
 A default partition to catch data outside the existing child boundaries is automatically created for all partition sets. The `check_default()` function provides monitoring for any data getting inserted into the default table and the `partition_data_`* set of functions can easily partition that data for you if it is valid data. That is much easier than automatically creating new child tables on demand and having to clean up potentially hundreds or thousands of unwanted partitions. And also better than throwing an error and losing the data!
 
@@ -22,6 +24,7 @@ For this extension, most of the attributes of the child partitions are all obtai
 | non-partition column primary key                  |                       |  14+                  |
 | non-partition column unique index                 |                       |  14+                  |
 | non-partition column unique index tablespace      |                       |  14+                  |
+| relation-specific options (autovac, etc)          |                       |  14+
 | unlogged table state*                             |                       |  14+                  |
 | non-unique indexes                                | 14+                   |                       |
 | privileges/ownership                              | 14+                   |                       |
@@ -150,8 +153,8 @@ RETURNS boolean
  * `p_control` - the column that the partitioning will be based on. Must be a time or integer based column
  * `p_interval` - the time or integer range interval for each partition. No matter the partitioning type, value must be given as text.
     + *\<interval\>*      - Any valid value for the interval data type. Do not type cast the parameter value, just leave as text.
-    + *\<integer\>*       - For ID based partitions, the integer value range of the ID that should be set per partition. Enter this as an integer in text format ('100' not 100). Currently must be greater than or equal to 2.
- * `p_type` - the type of partitioning to be done. Currently only **range** is supported.
+    + *\<integer\>*       - For ID based partitions, the integer value range of the ID that should be set per partition. Enter this as an integer in text format ('100' not 100). If the interval is >=2, then the `p_type` must be `range`. If the interval equals 1, then the `p_type` must be `list`. Also note that while numeric values are supported for id-based partitioning, the interval must still be a whole number integer.
+ * `p_type` - the type of partitioning to be done. Currently only **range** and **list** are supported. See `p_interval` parameter for special conditions concerning type.
  * `p_epoch` - tells `pg_partman` that the control column is an integer type, but actually represents and epoch time value. Valid values for this option are: 'seconds', 'milliseconds', 'nanoseconds', and 'none'. The default is 'none'. All table names will be time-based. In addition to a normal index on the control column, be sure you create a functional, time-based index on the control column (to_timestamp(controlcolumn)) as well so this works efficiently.
  * `p_premake` - is how many additional partitions to always stay ahead of the current partition. Default value is 4. This will keep at minimum 5 partitions made, including the current one. For example, if today was Sept 6th, and `premake` was set to 4 for a daily partition, then partitions would be made for the 6th as well as the 7th, 8th, 9th and 10th. Note some intervals may occasionally cause an extra partition to be premade or one to be missed due to leap years, differing month lengths, etc. This usually won't hurt anything and should self-correct (see **About** section concerning timezones and non-UTC). If partitioning ever falls behind the `premake` value, normal running of `run_maintenance()` and data insertion should automatically catch things up.
  * `p_start_partition` - allows the first partition of a set to be specified instead of it being automatically determined. Must be a valid timestamp (for time-based) or positive integer (for id-based) value. Be aware, though, the actual parameter data type is text. For time-based partitioning, all partitions starting with the given timestamp up to CURRENT_TIMESTAMP (plus `premake`) will be created. For id-based partitioning, only the partition starting at the given value (plus `premake`) will be made. Note that for subpartitioning, this only applies during initial setup and not during ongoing maintenance.
@@ -160,7 +163,7 @@ RETURNS boolean
  * `p_constraint_cols` - an optional array parameter to set the columns that will have additional constraints set. See the **About** section above for more information on how this works and the **apply_constraints()** function for how this is used.
  * `p_template_table` - If you do not pass a value here, a template table will automatically be made for you in same schema that pg_partman was installed to. If you pre-create a template table and pass its name here, then the initial child tables will obtain these properties discussed in the **About** section immediately.
  * `p_jobmon` - allow `pg_partman` to use the `pg_jobmon` extension to monitor that partitioning is working correctly. Defaults to TRUE.
- * `p_date_trunc_interval` - By default, pg_partman's time-based partitioning will truncate the child table starting values to line up at the beginning of typical boundaries (midnight for daily, day 1 for monthly, Jan 1 for yearly, etc). If a custom time interval that does not fall on those boundaries is desired, this option may be required to ensure the child table has the expected boundaries (especially if you also set p_start_partition). The valid values allowed for this parameter are the interval values accepted by the built-in date_trunc() function (day, week, month, etc). For example, if you set a 9-week interval, by default pg_partman would truncate the tables by month (since the interval is greater than one month but less than 1 year) and unexpectedly start on the first of the month in some cases. Set this value to week, so that the child table start values are properly truncated on a weekly basis to line up with the 9-week interval. If you are using a custom time interval, please experiment with this option to get the expected set of child tables you desire or use a more typical partitioning interval to simplify partition management.
+ * `p_date_trunc_interval` - By default, pg_partman's time-based partitioning will truncate the child table starting values to line up at the beginning of typical boundaries (midnight for daily, day 1 for monthly, Jan 1 for yearly, etc). If a partitioning interval that does not fall on those boundaries is desired, this option may be required to ensure the child table has the expected boundaries (especially if you also set `p_start_partition`). The valid values allowed for this parameter are the interval values accepted by PostgreSQL's built-in `date_trunc()` function (day, week, month, etc). For example, if you set a 9-week interval, by default pg_partman would truncate the tables by month (since the interval is greater than one month but less than 1 year) and unexpectedly start on the first of the month in some cases. Set this parameter value to `week`, so that the child table start values are properly truncated on a weekly basis to line up with the 9-week interval. If you are using a custom time interval, please experiment with this option to get the expected set of child tables you desire or use a more typical partitioning interval to simplify partition management.
 
 ```sql
 create_sub_parent(
@@ -665,7 +668,9 @@ Stores all configuration data for partition sets mananged by the extension.
     , constraint_valid boolean DEFAULT true NOT NULL
     , subscription_refresh text
     , ignore_default_data boolean NOT NULL DEFAULT true
-
+    , maintenance_order int DEFAULT NULL
+    , retention_keep_publication boolean NOT NULL DEFAULT false
+    , maintenance_last_run timestamptz
 
  - `parent_table`
     - Parent table of the partition set
@@ -696,6 +701,9 @@ Stores all configuration data for partition sets mananged by the extension.
  - `retention_keep_table`
     - Boolean value to determine whether dropped child tables only detached or actually dropped.
     - Default is TRUE to keep the table and only uninherit it. Set to FALSE to have the child tables removed from the database completely.
+ - `retention_keep_publication`
+    - If `retention_keep_table` is set to true, determines whether to drop the table from any publications that it may be a member of.
+    - Boolean value that defaults to false, meaning that by default tables that are not completely dropped as part of retention are removed from their publications.
  - `epoch`
     - Flag the table to be partitioned by time by an integer epoch value instead of a timestamp. See `create_parent()` function for more info. Default 'none'.
  - `constraint_cols`
@@ -718,8 +726,20 @@ Stores all configuration data for partition sets mananged by the extension.
     - Sets whether to inherit the ownership/privileges of the parent table to all child tables. Defaults to false and should only be necessary if you need direct access to child tables, by-passing the parent table.
   - `constraint_valid`
     - Boolean value that allows the additional constraints that pg_partman can manage for you to be created as NOT VALID. See "Constraint Exclusion" section at the beginning for more details on these constraints. This can allow maintenance to run much quicker on large partition sets since the existing data is not validated before additing the constraint. Newly inserted data is validated, so this is a perfectly safe option to set for data integrity. Note that constraint exclusion WILL NOT work until the constraints are validated. Defaults to true so that constraints are created as VALID. Set to false to set new constraints as NOT VALID.
-  - `subscription_refresh` - Name of a logical replication subscription to refresh when maintenance runs. If the partition set is subscribed to a publication that will be adding/removing tables and you need your partition set to be aware of these changes, you must name that subscription with this option. Otherwise the subscription will never become aware of the new tables added to the publisher unless you are refreshing the subscription via some other means. See the PG documentation for ALTER SUBSCRIPTION for more info on refreshing subscriptions - https://www.postgresql.org/docs/current/sql-altersubscription.html
- - `ignore_default_data` - By default, maintenance will ignore data in the default table when determining whether a new child table should be made. This means that if data is in the default and new child table would contain that data, an error will be thrown. If you need maintenance to acknowledge data in the default to fix a maintenance issue, this can be set to false. Note this can cause gaps in child table coverage, which can made data going into the default even worse, so it should not be left enabled once maintenance issues have been fixed.
+  - `subscription_refresh`
+    - Name of a logical replication subscription to refresh when maintenance runs. If the partition set is subscribed to a publication that will be adding/removing tables and you need your partition set to be aware of these changes, you must name that subscription with this option. Otherwise the subscription will never become aware of the new tables added to the publisher unless you are refreshing the subscription via some other means. See the PG documentation for ALTER SUBSCRIPTION for more info on refreshing subscriptions - https://www.postgresql.org/docs/current/sql-altersubscription.html
+ - `ignore_default_data`
+    - By default, maintenance will ignore data in the default table when determining whether a new child table should be made. This means that if data is in the default and new child table would contain that data, an error will be thrown. If you need maintenance to acknowledge data in the default to fix a maintenance issue, this can be set to false. Note this can cause gaps in child table coverage, which can made data going into the default even worse, so it should not be left enabled once maintenance issues have been fixed.
+ - `maintenance_order`
+    - Integer value that determines the order that maintenance will run the partition sets. Will run sets in increasing numerical order.
+    - Default value is NULL. All partition sets set to NULL will run after partition sets with a value defined. NULL partition sets run in an indeterminate order.
+    - For sub-partitioned sets, the child tables by default inherit the order of their parents. Child parent tables will run in logical order when their parent table's maintenance is run if left to the default value.
+ - retention_keep_publication
+    - If `retention_keep_table` is set to true so that tables are not fully dropped during retention, they will by default be removed from any publication that they are a part of. If you'd like to keep detached tables as part of the old partition set's publication, set this to true.
+    - Default value is false
+ - maintenance_last_run
+    - Timestamp of the last successful run of maintenance for this partition set. Can be useful as a monitoring metric to ensure partition maintenance is running properly.
+
 
 **`part_config_sub`**