Ares global data

From: Ty Heath <heath_at_mediadefender.com>
Date: Thu, 2 Aug 2007 17:50:03 -0700

This is a little FYI.

I have a log cap for each media type. In particular each of my data collectors will only log so many sources per minute. This is to avoid overwhelming the one database that it is sharing with other networks/devs.

This throttle is making both the video and audio sources hover at around 17 million per day.

I think the true number of video sources are about 30+ million unique entries per day, and the audio is upwards of 100+ million plus.

The database is saturated, meaning that if you ever want anything approaching real global data we'll need a dedicated database for it.

The current database is recording a total of about 100 Gigabytes per day, which is reasonable performance I think. Ares is using about 20% of that. The rest is going to gnutella mi supply I guess because of the massive redundancy in the way they are logging the data. We can possibly log more Ares data depending on how they clean up the Gnutella system.

Ty
Received on Fri Sep 14 2007 - 10:56:00 BST

This archive was generated by hypermail 2.2.0 : Sun Sep 16 2007 - 22:19:47 BST