Release Notes - Spark - Version 2.4.2 - HTML format

Bug

  • [SPARK-26961] - Found Java-level deadlock in Spark Driver
  • [SPARK-26998] - spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode
  • [SPARK-27216] - Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
  • [SPARK-27244] - Redact Passwords While Using Option logConf=true
  • [SPARK-27267] - Snappy 1.1.7.1 fails when decompressing empty serialized data
  • [SPARK-27275] - Potential corruption in EncryptedMessage.transferTo
  • [SPARK-27301] - DStreamCheckpointData failed to clean up because it's fileSystem cached
  • [SPARK-27338] - Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator
  • [SPARK-27351] - Wrong outputRows estimation after AggregateEstimation with only null value column
  • [SPARK-27390] - Fix package name mismatch
  • [SPARK-27394] - The staleness of UI may last minutes or hours when no tasks start or finish
  • [SPARK-27403] - Fix `updateTableStats` to update table stats always with new stats or None
  • [SPARK-27406] - UnsafeArrayData serialization breaks when two machines have different Oops size
  • [SPARK-27419] - When setting spark.executor.heartbeatInterval to a value less than 1 seconds, it will always fail
  • [SPARK-27453] - DataFrameWriter.partitionBy is Silently Dropped by DSV1

Improvement

  • [SPARK-27346] - Loosen the newline assert condition on 'examples' field in ExpressionInfo
  • [SPARK-27358] - Update jquery to 1.12.x to pick up security fixes
  • [SPARK-27479] - Hide API docs for "org.apache.spark.util.kvstore"

Task

  • [SPARK-27382] - Update Spark 2.4.x testing in HiveExternalCatalogVersionsSuite

Edit/Copy Release Notes

The text area below allows the project release notes to be edited and copied to another document.