I am running pyspark commands is jupyter notebook.
Whenever I disconnect the dataframe created is lost . Why is it so?
Do the dataframes exist till the session is active ?
Yes, that’s right.
Dataframes and RDDs are like the variable definition in any programming language. They remain in memory as long as the process is running and the variables are in scope.