Here it is using Scala DataFrame functions: from_unixtime and to_date
// NOTE: divide by 1000 required if milliseconds
// e.g. 1446846655609 -> 2015-11-06 21:50:55 -> 2015-11-06
mr.select(to_date(from_unixtime($"ts" / 1000)))
Related Contents:
- How to store custom objects in Dataset?
- Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
- Write to multiple outputs by key Spark – one Spark job
- How to convert rdd object to dataframe in spark
- Spark 2.0 Dataset vs DataFrame
- Write single CSV file using spark-csv
- How to define and use a User-Defined Aggregate Function in Spark SQL?
- How to load local file in sc.textFile, instead of HDFS
- Automatically and Elegantly flatten DataFrame in Spark SQL
- How do I detect if a Spark DataFrame has a column
- Case class equality in Apache Spark
- Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
- How to define schema for custom type in Spark SQL?
- Flattening Rows in Spark
- Split 1 column into 3 columns in spark scala
- Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
- How to pass -D parameter or environment variable to Spark job?
- Spark unionAll multiple dataframes
- How can I pass extra parameters to UDFs in Spark SQL?
- How to split a dataframe into dataframes with same column values?
- How to use regex to include/exclude some input files in sc.textFile?
- How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
- Append a column to Data Frame in Apache Spark 1.3
- How do I convert csv file to rdd
- How to aggregate values into collection after groupBy?
- Defining a UDF that accepts an Array of objects in a Spark DataFrame?
- Spark / Scala: forward fill with last observation
- How to define a custom aggregation function to sum a column of Vectors?
- Customize SparkContext using sparkConf.set(..) when using spark-shell
- Spark dataframe write method writing many small files
- Spark losing println() on stdout
- About how to add a new column to an existing DataFrame with random values in Scala
- Renaming column names of a DataFrame in Spark Scala
- How to find spark RDD/Dataframe size?
- Reading DataFrame from partitioned parquet file
- How to save DataFrame directly to Hive?
- Spark DataFrame: does groupBy after orderBy maintain that order?
- Stackoverflow due to long RDD Lineage
- Change nullable property of column in spark dataframe
- Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]
- How to use COGROUP for large datasets
- How to save a spark DataFrame as csv on disk?
- What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
- Modify collection inside a Spark RDD foreach
- Explanation of fold method of spark RDD
- Spark – Error “A master URL must be set in your configuration” when submitting an app
- How to write unit tests in Spark 2.0+?
- How to compare two dataframe and print columns that are different in scala
- Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
- Spark DataFrames when udf functions do not accept large enough input variables