[SEDONA-655] DBSCAN

[james-willis]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide

Is this PR related to a JIRA ticket?

• Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-655. The PR name follows the format [SEDONA-XXX] my subject.

What changes were proposed in this PR?

This PR add a DBSCAN function in the scala and python APIs of the spark implementation of sedona.

How was this patch tested?

unit tests

Did this PR include necessary documentation updates?

TODO

- Yes, I am adding a new API. I am using the current SNAPSHOT version number in vX.Y.Z format.

[jiayuasu]

@james-willis The memory consumption of new tests seem to be very high?

[james-willis]

@james-willis The memory consumption of new tests seem to be very high?

This is something I had worked with Kristin on in the past. We had thought it was related to cached dataframes holding references to broadcast relations. But this committed version of the code does not contain any caching, just checkpoints.

There is some persisting inside of the connected component implementation, but those should be getting cleaned up. I'll check if something here is leaking persisted dataframes

[james-willis]

Theres some dataframes inside of the connected components algo that dont get unpersisted. I will look into raising a PR in that repo tomorrow to fix that.

[james-willis]

I submitted a fix to graphframes for this. I can clear the spark catalog if the disable broadcast fix doesnt resolve

graphframes/graphframes#459