Writing UDF for looks up in the Map in java giving Unsupported literal type class java.util.HashMap

by

You can pass the look up map or array etc. to the udf by using partial. check out this example.

from functools import partial
from pyspark.sql.functions import udf

fruit_dict = {"O": "Orange", "A": "Apple", "G": "Grape"}
df = spark.createDataFrame([("A", 20), ("G", 30), ("O", 10)], ["Label", "Count"])
def decipher_fruit(label, fruit_map):
  label_names = list(fruit_map.keys())
  if label in label_names:
    return fruit_map[label]
  return None

decipher_fruit_udf = udf(partial(decipher_fruit, fruit_map = fruit_dict), StringType())
df2 = df.withColumn("fruit_name", decipher_fruit_udf("label"))
display(df2)

Leave a Comment