Pyspark: explode json in column to multiple columns
Jun 28, 2018 · Pyspark: explode json in column to multiple columns Asked 7 years ago Modified 3 months ago Viewed 86k times
pyspark - How to use AND or OR condition in when in Spark
105 pyspark.sql.functions.when takes a Boolean Column as its condition. When using PySpark, it's often useful to think "Column Expression" when you read "Column". Logical operations on …
PySpark: How to fillna values in dataframe for specific columns?
Jul 12, 2017 · PySpark: How to fillna values in dataframe for specific columns? Asked 7 years, 11 months ago Modified 6 years, 2 months ago Viewed 200k times
pyspark dataframe filter or include based on list
Nov 4, 2016 · I am trying to filter a dataframe in pyspark using a list. I want to either filter based on the list or include only those records with a value in the list. My code below does not work: # …
PySpark: How to Append Dataframes in For Loop - Stack Overflow
May 29, 2019 · PySpark: How to Append Dataframes in For Loop Asked 6 years, 1 month ago Modified 2 years, 10 months ago Viewed 43k times
How to find count of Null and Nan values for each column in a …
Jun 19, 2017 · How to find count of Null and Nan values for each column in a PySpark dataframe efficiently? Asked 8 years ago Modified 2 years, 3 months ago Viewed 287k times
pyspark - Apache spark dealing with case statements - Stack …
Oct 11, 2016 · I am dealing with transforming SQL code to PySpark code and came across some SQL statements. I don't know how to approach case statments in pyspark? I am planning on …
Show distinct column values in pyspark dataframe - Stack Overflow
With pyspark dataframe, how do you do the equivalent of Pandas df['col'].unique(). I want to list out all the unique values in a pyspark dataframe column. Not the SQL type way …
How to create a copy of a dataframe in pyspark? - Stack Overflow
Sep 12, 2018 · To create a Deep copy of a PySpark DataFrame, you can use the rdd method to extract the data as an RDD, and then create a new DataFrame from the RDD. df_deep_copied …
Python/pyspark data frame rearrange columns - Stack Overflow
Python/pyspark data frame rearrange columns Asked 8 years, 3 months ago Modified 3 years ago Viewed 109k times