@sandeepgiri @abhinav As explained in session, I want to know detail regarding the Onfly calculations.
I have and periodic batch job computing the data and loading into database.
Apart from this i have to perform calculations on fly from an web application based to the user requested params.
So how do i perform these calculations
Hi Krishnakanth,
Good Question!
As per my understanding, here, on-the-fly computation refers to the computation needed by a request on a web application.
Here computations can take huge time. These computations can be done on-fly as long the overall response time is acceptable. There are many innovations done in order to reduce the overall response time. You would not want the overall response time to be very high.
There are many ways to reduce the overall response time. Here are some of those
1. Pre-computation with Batch Processing
This is the most common way of doing the heavy lifting. We prepare the data using a separate program which might be running on Spark or Hadoop. This separate program is run independent of the user and thus prepares the data or does the computation for all users.
Most of the recommendation systems are designed this way.
2. Caching
A lot of time when certain computations take too much time. Instead of computing again, we start using caching. Memcached is probably the best solution as of now.
3. Delegating to other services
This is another very good architecture followed in high traffic website. Basically, the web application assembles the result received from many services. Since calls to all the services are made in parallel, the overall time taken is equal to the slowest service not equal to the total of all services. Though it does increase an overhead of a network transfer which is generally small if the connections are cached and DNS lookups are avoided.
4. In-memory computation such as Redis
If your computation is heavily in-memory, you can use a system such Redis - an in-memory NOSQL to perform heavy lifting.
5. Chunked Transfer / Response
At times, your web service can give the results but in parts, in those scenerios you could use the chunked transfer.
6. Socket IO
At times, you may want to show the progress of a process that takes too long to compute. In such scenarios, you could use either the chunked transfer or Socket IO kind of library.
Your solution would be a combination of the above approaches based the problem at hand.