Counting records with changing stages


#1

Hello All,

I have a scenario in which i am getting records from users like -:

Userid,Name,Territory,country,StageID

So the records are coming in a stream which i am getting from kafka.

The scenario is like 1st record i get is like-:

12,Sachin,Delhi,Delhi,1

again if the record is updated then it is pushed again in kafka and i get-:

12,Sachin,Delhi,Delhi,2

So what i need to do is i need to count the records in the basis of stages and if a particular record moves from 1 stage to other then that stage should reflect 1 less record and other stage would be reflecting 1 more record.

Kindly help on this.

Till now i have seen spark structured streaming and which seems to solve the above problem. So is there anything else which can solve the above scenario.

Regards,
Sachin