MoNA 2022 Spring 2.6.3 Update and Postgresql Data Migration

• Removed MongoDB and ElasticSearch Dual Database System
• Created Postgresql Database
  • In efforts to even out our codebase, the team has decided to utilize Postgresql across all our applications.
  • The Springboot upgrade was the best time to additionally refactor all the code to use Postgresql rather then attempt to update mongoDB and ElasticSearch which was proving difficult to maintain.
• Newly Designed Spring Data JPA Schema for Interaction with Postgresql
  • Spring Data JPA is a powerful library that allows for easy creation of the Postgresql database schema and easy ways to store and retrieve our data
• Upgraded All Core Functionalities to Spring Boot 2.6.3
  • MoNA was very behind on Springboot versions (1.4.5 to be exact) and the IT team is striving to update all our applications to the latest codebase
• Removed Spring Cloud Netflix Zuul from Proxy Service
• Implemented Newly Designed Spring Cloud Gateway for Proxy Service
  • Spring Cloud Gateway was necessary as Zuul has been deprecated from the latest Springboot versions
  • As a result all routes have been completely rewritten but should work just as zuul did before
• Separated Discovery Service and Config Service to be independent services
  • Having the discovery service handle discovery duties and act as the config service at the same time caused many issues with the Springboot update so it made sense to separate these two
• Fixed All Scala Tests to Pass
  • This does not include web hooks but that crosses against production which is not up at the time of writing this
• Completely Refactored Statistics Module for use with Postgresql Database
  • Prior the statistics module utilized mongoDB operations which were not optimized in the currently used version, which led to the statistics module constantly crashing or never even finishing
  • Now, the statistics module utilizes Java Streams to fetch all the data and all javascript embedded code has been translated to proper Scala code which should allow statistics to always run and finish
• Refactored All Core Modules and Services to use new Spring Data JPA Schema
  • In current implementation, objects are simply that of Scala Objects that have been mapped out
  • Now, all services and modules interact with the persistence layer through the JPA, utilizing proper get and set methods.
• Removed RSQL search system
• Implemented Spring Filtering system that retains RSQL search ability but works with Spring Data JPA
  • MoNA system heavily uses queries to filter the data for services like Persistence, Downloads, Similarity and is used by LC Binbase
  • See library here https://github.com/turkraft/spring-filter
• Similarity Service now loads faster and has an API to call when reload is needed. Restarting the entire application is no longer needed just to load the similarity service.
  • Utilize Java Streams to quickly and efficiently supply data to similarity service without overusing memory
• Fixed Docker Compose files
  • Small modifications needed to create new Postgresql Database
  • Remove Containers for MongoDB and ElasticSearch
  • Add Config Service
  • Tweak various variables for memory allocation
• Removed Old Unused code from repository
• Removed Springfox Swagger-UI
• Implemented Spring Docs OpenAPI with Swagger-UI
  • Springfox was also deprecated and did not work well with the latest Swagger-UI changes so Spring Docs OpenAPI was the logical thing to implement
• Added new Adduct to adducts list
• LipidBlast 2022 Library will now be tagged with 'In-Silico'
  • Fixes issue: https://github.com/metabolomics-us/mona/issues/65
• Added new Documentation to Github Wiki Pages
• Added new Documentation for new Spring Filter and cURL querying systems
• Various Angular fixes to support new or changed APIs
• Fixed Querying system to use Spring Filter
• Added more error messages to Advanced Uploader when a file cannot be parsed
• Added New Password Requirements for new Accounts
• Added New Password Reset Functionality for Admins
• Fixed cURL statement that was generated during Spectra Browser queries. Should work with simple copy paste now.
• Enhanced Various Error Messages
• Created more models for use in Angular so that frontend code takes more advantage of typescripts type inference functionalities.
  • More work needs to be done here to take use of these new models

Condense Space Usage on Similarity Service

The Calculate All Adducts feature required metaData from the Compound object in order to search for the adducts in the Similarity service. The consequence of adding the Compounds object was that it added a significant amount of data to the SimpleSpectrum objects that are stored in memory in the Similarity Service. This caused multiple java memory heap errors on large requests. To circumvent this problem, we specifically parse the Compound for Theoretical Adducts and store the results in an Array of Doubles. This should restore virtually restore the SimpleSpectrum objects to their original size and avoid java heap memories. No changes will be needed to the API requests or requests format.
Additionally, changed the docker base image from Java(which was deprecated) to openjdk:8.

FIEHNLAB-1820/Calculate all possible adducts for spectra

• Curation Runner has new process to calculate theoretical adducts. These are theoretical adducts are used for better similarity prediction with LC Binbase
• Similarity service now has a new request parameter "checkAllAdducts", this is a boolean that when set to true will try to match the user provided precursorMz with all calculated theoretical adducts
• Fixed bug in frontend where submitter.id was not persisting when a user uploaded spectra

FIEHNLAB-1783/Fix Null Pointer Exception and Cap Log Sizes

• Remove InChI code from downloads that was causing null pointer exception
• Capped log files so they no longer fill up entire server

FIEHNLAB-1780/Fix Downloads

• Change pick size from 25 to 250 to increase efficiency
• Changed CRON jobs to create files once a week on SAT and WED

Closes https://github.com/metabolomics-us/mona/issues/49

Emergency Fix: SMILES Adding to File Even When Not Present on Upload File

• Fixed an issue where SMILES metadata was being added to the upload payload even when it wasn't present on the original upload file

FIEHNLAB-1649/AddInChIToDownloadFiles,FIEHNLAB-375/FixSubmitterCountInMonaStatistics,FIEHNLAB-1675/AddEntropySimilarity,FIEHNLAB-1680/FixUploadProcessing

• Adding InChI to download file since it is a more dependable identifier
• Fix the submitter count to be accurate for the statistics
• Added Entropy Similarity for the Similarity Service
• Added ability to select different similarity algorithms when requesting similarity search
• Fixed various bugs related to the upload process in MoNA

Fix Similarity Service to work with removePrecursorIon for BinBase

• Fixing Similarity Service to properly work with removePrecursorIon feature
• Fixed frontend Similarity section on the spectra/display page
• Added documentation on how to start, run, develop, and deploy the MoNA application
• Made some changes to fix deployment and use alternate cts per environment if needed