Fix PartitionedFile.filePath handling for spark 3.5.0 #114
Conversation
|
Kudos, SonarCloud Quality Gate passed!
|
| @@ -36,7 +36,7 @@ lazy val commonIOVersion = "2.5" | |||
| lazy val logbackVersion = "1.1.7" | |||
| lazy val scoptVersion = "3.7.1" | |||
| lazy val akkaVersion = "2.5.31" | |||
| lazy val spark3Version = "3.1.1" | |||
| lazy val spark3Version = "3.5.0" | |||
There was a problem hiding this comment.
Does this change break compatibility with Spark 3.2 and 3.1?
At the moment, I can not expend time doing the testing (basically, I'm focused on looking for a new contract. If you know one, let me know. 😉 )
There was a problem hiding this comment.
I will do some testing with spark 3.3, 3.2, 3.1 and will let you know about the result.
If I see an interesting project, I will let you know.
There was a problem hiding this comment.
Any progress on this? You could keep the version on 3.1.1 and just call toString anyway - for people using Spark 3.4+ it will call toString() on the SparkPath returned - for people using older versions nothing will change?
There was a problem hiding this comment.
Actually the fix is not working for older versions.
There was a problem hiding this comment.
Would it be possible to increase the version, and release it for Spark 3.5?
There was a problem hiding this comment.
@jensgoossens-tomtom @alexeisolodovnicov-tomtom Maybe your company is able to support this project. That would give me a little bit of energy and motivation. Even one beer would be enough. ;)
I'm open to any type of collaboration. Are they looking for Solution Architects?
Fixes #113