Lossy Count Algorithm

The lossy count algorithm is an algorithm to identify elements in a data stream whose frequency count exceed a user-given threshold. The frequency computed by this algorithm is not always accurate, but has an error threshold that can be specified by the user. The run time space required by the algorithm is inversely proportional to the specified error threshold, hence larger the error, the smaller the footprint. It was created by eminent computer scientists Rajeev Motwani and Gurmeet Singh Manku. This algorithm finds huge application in computations where data takes the form of a continuous data stream instead of a finite data set, for e.g. network traffic measurements, web server logs, clickstreams.

Algorithm

The algorithm followed is outlined as follow

• Step 1: Divide the incoming data stream into buckets of width $w=1/\epsilon$, where $\epsilon$ is mentioned by user as the error bound (along with minimum support threshold = $\sigma$).
• Step 2: Increment the frequency count of each item according to the new bucket values. After each window, decrement all counters by 1.
• Step 3: Repeat – Update counters and after each bucket, decrement all counters by 1.

References

• Motwani, R; Manku, G.S (2002). "Approximate frequency counts over data streams". VLDB '02 Proceedings of the 28th international conference on Very Large Data Bases: 346–357. {{cite journal}}: Invalid |ref=harv (help)