scala.util.Random.nextFloat() —> every time we run it gives fractional value from 0 to 1
val x=sc.parallelize(1 to 10)
var requried_fraction=0.5
def retain(x:Int):Boolean=
{
scala.util.Random.nextFloat() <= requried_fraction
}
when we pass this function to filter like
val sample_rdd=x.filter(retain)
question: here every element of input rdd goes through the filter function , some will be retained and some are not retained. but where exactly we are limiting the o/p to required fraction (i mean if unluckily 10 elements retained by this condition (scala.util.Random.nextFloat() <= requried_fraction) which is not expected .
are all elements will not traverse through the filter /even though it traversed how it knows it got enough samples???