Spark Dataframe validating column names for parquet writes

For everyone experiencing this in pyspark: this even happened to me after renaming the columns. One way I could get this to work after some iterations is this:

file = "/opt/myfile.parquet"
df = spark.read.parquet(file)
for c in df.columns:
    df = df.withColumnRenamed(c, c.replace(" ", ""))

df = spark.read.schema(df.schema).parquet(file)

Related Contents:

  1. How to save/insert each DStream into a permanent table
  2. How to melt Spark DataFrame?
  3. Using a column value as a parameter to a spark DataFrame function
  4. Find maximum row per group in Spark DataFrame
  5. Unpivot in spark-sql/pyspark
  6. Split Spark Dataframe string column into multiple columns
  7. How to access element of a VectorUDT column in a Spark DataFrame?
  8. How to check if spark dataframe is empty?
  9. Avoid performance impact of a single partition mode in Spark window functions
  10. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  11. pyspark dataframe filter or include based on list
  12. Pyspark: Pass multiple columns in UDF
  13. Pyspark : forward fill with last observation for a DataFrame
  14. How to loop through each row of dataFrame in pyspark
  15. PySpark: how to resample frequencies
  16. spark.ml StringIndexer throws ‘Unseen label’ on fit()
  17. Rename more than one column using withColumnRenamed
  18. Why does Spark think this is a cross / Cartesian join
  19. Keep only duplicates from a DataFrame regarding some field
  20. More than one hour to execute pyspark.sql.DataFrame.take(4)
  21. pyspark: count distinct over a window
  22. Convert pyspark string to date format
  23. Spark Equivalent of IF Then ELSE
  24. Multiple Aggregate operations on the same column of a spark dataframe
  25. Partitioning in spark while reading from RDBMS via JDBC
  26. How to change a dataframe column from String type to Double type in PySpark?
  27. How to control partition size in Spark SQL
  28. How do I add a new column to a Spark DataFrame (using PySpark)?
  29. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  30. Does spark predicate pushdown work with JDBC?
  31. Explode in PySpark
  32. Spark DataFrame: count distinct values of every column
  33. Create Spark DataFrame. Can not infer schema for type
  34. GroupBy column and filter rows with maximum value in Pyspark
  35. Passing a data frame column and external list to udf under withColumn
  36. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  37. Dividing complex rows of dataframe to simple rows in Pyspark
  38. How to fix ‘TypeError: an integer is required (got type bytes)’ error when trying to run pyspark after installing spark 2.4.4
  39. How do I add an persistent column of row ids to Spark DataFrame?
  40. Apache spark dealing with case statements
  41. Spark DataFrame: does groupBy after orderBy maintain that order?
  42. Cast column containing multiple string date formats to DateTime in Spark
  43. How to explode multiple columns of a dataframe in pyspark
  44. How to pivot on multiple columns in Spark SQL?
  45. How to group by common element in array?
  46. Using UDF ignores condition in when
  47. Create single row dataframe from list of list PySpark
  48. Cannot find col function in pyspark
  49. Spark specify multiple column conditions for dataframe join
  50. What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?

Leave a Comment