[Spark] 에러 - You can set "spark.sql.legacy.timeParserPolicy" to "LEGACY" to restore the behavior before Spark 3.0, or set to "CORRECTED" and treat it as an invalid datetime string.

에러 내용

Caused by: org.apache.spark.SparkUpgradeException: 
[INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] 
You may get a different result due to the upgrading to Spark >= 3.0:
Fail to parse '2024-05-26 12:11:38.478' in the new parser. 
You can set "spark.sql.legacy.timeParserPolicy" to "LEGACY" to restore 
the behavior before Spark 3.0, or set to "CORRECTED" 
and treat it as an invalid datetime string.

 

해결 방법

spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")