diff --git a/DESCRIPTION b/DESCRIPTION index d2c65392..aee1eb0f 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,8 +1,8 @@ Package: DataQualityDashboard Type: Package Title: Execute and View Data Quality Checks on OMOP CDM Database -Version: 2.3.0 -Date: 2023-05-21 +Version: 2.4.0 +Date: 2023-07-26 Authors@R: c( person("Katy", "Sadowski", email = "sadowski@ohdsi.org", role = c("aut", "cre")), person("Clair", "Blacketer", role = c("aut")), diff --git a/NEWS.md b/NEWS.md index 3e3e0d1b..62533c1f 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,23 @@ +DataQualityDashboard 2.4.0 +========================== +This release includes: + +### Threshold file updates +**The following changes involve updates to the default data quality check threshold files. If you are currently using an older version of DQD and update to v2.4.0, you may see changes in your DQD results. The failure threshold changes are fixes to incorrect thresholds in the v5.4 files and thus should result in more accurate, easier to interpret results. The unit concept ID changes ensure that long-invalid concepts will no longer be accepted as plausible measurement units.** + +- The incorrect failure thresholds for `measurePersonCompleteness` and `measureValueCompleteness` were fixed in the v5.4 table & field level threshold files. This issue has existed since v5.4 support was initially added in March 2022 + - Many `measurePersonCompleteness` checks had a threshold of 0 when it should have been 95 or 100 + - Many `measureValueCompleteness` checks had a threshold of 100 when it should have been 0, and many had no threshold (defaulting to 0) when it should have been 100 + - The thresholds have now been updated to match expectations for required/non-required tables/fields +- In the v5.2, v5.3, and v5.4 table level threshold files, `measurePersonCompleteness` for the DEATH table has been toggled to `Yes`, with a threshold of 100 +- In the v5.2, v5.3, and v5.4 concept level threshold files, all references to unit concept 9117 in `plausibleUnitConceptIds` have been updated to 720870. Concept 9117 became non-standard and was replaced with concept 720870, on 28-Mar-2022 +- In the v5.2, v5.3, and v5.4 concept level threshold files, all references to unit concepts 9258 and 9259 in `plausibleUnitConceptIds` have been removed. These concepts were deprecated on 05-May-2022 + +### Bugfix +- Call to new function `convertJsonResultsFileCase` in Shiny app was appended with `DataQualityDashboard::`. This prevents potential issues related to package loading and function naming conflicts + +Some minor refactoring of testthat files and package build configuration and some minor documentation updates were also added in this release. + DataQualityDashboard 2.3.0 ========================== This release includes: diff --git a/R/convertResultsCase.R b/R/convertResultsCase.R index b6d4f357..0668b211 100644 --- a/R/convertResultsCase.R +++ b/R/convertResultsCase.R @@ -15,7 +15,7 @@ # limitations under the License. #' @title Convert JSON results file case -#' +#' #' @description Convert a DQD JSON results file between camelcase and (all-caps) snakecase. Enables viewing of pre-v.2.1.0 results files in later DQD versions, and vice versa #' #' @param jsonFilePath Path to the JSON results file to be converted diff --git a/README.md b/README.md index fdd1d407..dc3c21e8 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,16 @@ User Documentation ================== Documentation can be found on the [package website](https://ohdsi.github.io/DataQualityDashboard/index.html). +PDF versions of the documentation are also available: + +* Vignette: [Add a New Data Quality Check](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/AddNewCheck.pdf) +* Vignette: [Check Status Descriptions](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/CheckStatusDefinitions.pdf) +* Vignette: [Data Quality Check Type Descriptions](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/CheckTypeDescriptions.pdf) +* Vignette: [Running the DQD on a Cohort](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/DqdForCohorts.pdf) +* Vignette: [Failure Thresholds and How to Change Them](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/Thresholds.pdf) +* Vignette: [SqlOnly Mode](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/SqlOnly.pdf) +* Package manual: [DataQualityDashboard manual](https://github.com/OHDSI/DataQualityDashboard/raw/main/inst/doc/DataQualityDashboard.pdf) + Support ======= diff --git a/docs/404.html b/docs/404.html index 7188b296..08b81bfe 100644 --- a/docs/404.html +++ b/docs/404.html @@ -32,7 +32,7 @@
diff --git a/docs/LICENSE-text.html b/docs/LICENSE-text.html index 8a7fd530..e17e3b8a 100644 --- a/docs/LICENSE-text.html +++ b/docs/LICENSE-text.html @@ -17,7 +17,7 @@ diff --git a/docs/articles/AddNewCheck.html b/docs/articles/AddNewCheck.html index 5769a243..0deb430c 100644 --- a/docs/articles/AddNewCheck.html +++ b/docs/articles/AddNewCheck.html @@ -33,7 +33,7 @@ @@ -108,7 +108,7 @@vignettes/AddNewCheck.rmd
AddNewCheck.rmd
This will bring up a SqlRender window in your browser. Paste in your complete DQ query. The program will see the input parameters and allow you to define them. The rendered translation, dialect specific, is output. You can run the “Rendered Translation” on your OMOP CDM.
-SQL Server SQL is the lingua franca for Data Quality Dashboard queries. SQLRender will modify the query to other SQL dialects. See SQLRender diff --git a/docs/articles/CheckStatusDefinitions.html b/docs/articles/CheckStatusDefinitions.html index 5bf3ecd3..054f2894 100644 --- a/docs/articles/CheckStatusDefinitions.html +++ b/docs/articles/CheckStatusDefinitions.html @@ -33,7 +33,7 @@
vignettes/CheckStatusDefinitions.rmd
CheckStatusDefinitions.rmd
vignettes/CheckTypeDescriptions.rmd
CheckTypeDescriptions.rmd
vignettes/DataQualityDashboard.rmd
DataQualityDashboard.rmd
vignettes/DqdForCohorts.rmd
DqdForCohorts.rmd
vignettes/SqlOnly.rmd
SqlOnly.rmd
This article describes how to use DQD to generate only the SQL that -executes all DataQualityDashoard checks, without actually executing +executes all DataQualityDashboard checks, without actually executing them. There are a few main advantages of running DQD in Sql-only mode:
vignettes/Thresholds.rmd
Thresholds.rmd
::executeDqChecks(connectionDetails = connectionDetails,
- DataQualityDashboardcdmDatabaseSchema = cdmDatabaseSchema,
- resultsDatabaseSchema = resultsDatabaseSchema,
- cdmSourceName = cdmSourceName,
- numThreads = numThreads,
- sqlOnly = sqlOnly,
- outputFolder = outputFolder,
- verboseMode = verboseMode,
- writeToTable = writeToTable,
- checkLevels = checkLevels,
- tablesToExclude = tablesToExclude,
- checkNames = checkNames,
- tableCheckThresholdLoc = location of the table check file,
- fieldCheckThresholdLoc = location of the field check file,
- conceptCheckThresholdLoc = location of the concept check file)
DataQualityDashboard::executeDqChecks(connectionDetails = connectionDetails,
+ cdmDatabaseSchema = cdmDatabaseSchema,
+ resultsDatabaseSchema = resultsDatabaseSchema,
+ cdmSourceName = cdmSourceName,
+ numThreads = numThreads,
+ sqlOnly = sqlOnly,
+ outputFolder = outputFolder,
+ verboseMode = verboseMode,
+ writeToTable = writeToTable,
+ checkLevels = checkLevels,
+ tablesToExclude = tablesToExclude,
+ checkNames = checkNames,
+ tableCheckThresholdLoc = location of the table check file,
+ fieldCheckThresholdLoc = location of the field check file,
+ conceptCheckThresholdLoc = location of the concept check file)
Documentation can be found on the package website.
+PDF versions of the documentation are also available:
+NEWS.md
+ This release includes:
+The following changes involve updates to the default data quality check threshold files. If you are currently using an older version of DQD and update to v2.4.0, you may see changes in your DQD results. The failure threshold changes are fixes to incorrect thresholds in the v5.4 files and thus should result in more accurate, easier to interpret results. The unit concept ID changes ensure that long-invalid concepts will no longer be accepted as plausible measurement units.
+measurePersonCompleteness
and measureValueCompleteness
were fixed in the v5.4 table & field level threshold files. This issue has existed since v5.4 support was initially added in March 2022
+measurePersonCompleteness
checks had a threshold of 0 when it should have been 95 or 100measureValueCompleteness
checks had a threshold of 100 when it should have been 0, and many had no threshold (defaulting to 0) when it should have been 100measurePersonCompleteness
for the DEATH table has been toggled to Yes
, with a threshold of 100plausibleUnitConceptIds
have been updated to 720870. Concept 9117 became non-standard and was replaced with concept 720870, on 28-Mar-2022plausibleUnitConceptIds
have been removed. These concepts were deprecated on 05-May-2022convertJsonResultsFileCase
in Shiny app was appended with DataQualityDashboard::
. This prevents potential issues related to package loading and function naming conflictsSome minor refactoring of testthat files and package build configuration and some minor documentation updates were also added in this release.
+This release includes:
sqlOnly
and sqlOnlyIncrementalInsert
to TRUE in executeDqChecks
will return (but not run) a set of SQL queries that, when executed, will calculate the results of the DQ checks and insert them into a database table. Additionally, sqlOnlyUnionCount
can be used to specify a number of SQL queries to union for each check type, allowing for parallel execution of these queries and potentially large performance gains. See the SqlOnly vignette for detailssqlOnly
and sqlOnlyIncrementalInsert
to TRUE in executeDqChecks
will return (but not run) a set of SQL queries that, when executed, will calculate the results of the DQ checks and insert them into a database table. Additionally, sqlOnlyUnionCount
can be used to specify a number of SQL queries to union for each check type, allowing for parallel execution of these queries and potentially large performance gains. See the SqlOnly vignette for details
convertJsonResultsFileCase
can be used to convert the keys in a DQD results JSON file between snakecase and camelcase. This allows reading of v2.1.0+ JSON files in older DQD versions, and other conversions which may be necessary for secondary use of the DQD results file. See function documentation for details