RE: IEA Deck March 17th - 31st

From: Ben Grodsky <>
Date: Fri, 6 Apr 2007 14:23:00 -0700

Let me know when you have time to discuss this. Jay, Tabish and I talked through the current Universal Studios situation. It's great that they're giving us access to their testing database where they record each step, but:


                        Their sample sizes are too small: several of their tests are for 4, 2, and 0 attempts; while 20 is still a small set, it's a lot closer to be statistically significant.
                        They are not inputting accurate data into their database, from which they generate the PPT.
                        Their testing methodology still is inconsistent in practice with what they've said they do. This hurts us because we are NOT getting any credit for things that we can clearly see the effect of them (e.g., BT interdiction, our eMule servers)


                The data they're recording is inconsistent with reality.
                They've given us their server.mets which show servers with higher numbers than the ONE they connected on for all of their tests this testing period.
                This testing period they focused on BiG BanG 9, despite having larger servers in their server met and their purported methodology being that they connect to the server with the largest number of users and files (they sort by users, then they sort by files -- which is retarded anyway because the 2nd sort wipes out anything done by the 1st sort).
                This testing suggests that we've been contracted to protect only on BiG BanG 9 and doesn't reflect their stated methodology and doesn't give us credit for all the protection we are doing on other servers. After all, we protect on the whole network and don't focus everything just on the one BiG BanG 9 ed2k server.


                According to their database (because we can't tell from the PPT) they have some downloads that finished in the past (obviously not inputted properly), and the ones that were "Good Downloads" all coincidentally ended within minutes of each other.
                It's unclear that they are actually sticking to keeping their downloads <=24 hours. Their database suggests some of the tests ran days. Basically, do the download times even matter?!
                If would be extremely helpful for future troubleshooting, if they included the INFO HASH for the torrents they test.

                                This would help determine whether interdiction was active during that time period
                                and would help determine whether they're accidentally counting our decoys against us.


From: Randy Saaf
Sent: Fri 06-Apr-07 13:50
To: Tabish Hasan; qa; torrents
Subject: Re: IEA Deck March 17th - 31st

What about the black donnellys test?

----- Original Message -----
From: Tabish Hasan
To: Randy Saaf; qa; torrents
Sent: Fri Apr 06 10:42:57 2007
Subject: RE: IEA Deck March 17th - 31st

The results aren't too surprising. The weak networks were BT, eMule, and Ares. Gnutella looked good.

They're seeing low results on eMule because they're doing most of their tests on Big Bang 9 (as discussed earlier). For Ares, our protection is still not up fully, so they're pretty much seeing what we're seeing internally.

On BT, they're still doing most of their tests on torrentz / mininova / snarf-it. They did do some tests on torrentspy and torrentportal, but that was very limited. Torrents team, continue to focus getting our numbers higher on mininova and snarf-it.

I've uploaded the PPT sheet to \\mdfile01\public\QA\Universal Studios\2007_04_06 (Mar 17-31) <file:///\\mdfile01\public\QA\Universal%20Studios\2007_04_06%20(Mar%2017-31)> . I also uploaded their BT test data and data from all networks in case anyone wants to analyze their tests further. Note the sheets only show data starting from March 1st.


-----Original Message-----
From: Randy Saaf
Sent: Thursday, April 05, 2007 9:11 PM
To: qa; torrents
Subject: Fw: IEA Deck March 17th - 31st

----- Original Message -----

From: Markham, Aaron (NBC Universal) <>

To: Randy Saaf; Octavio Herrera

Sent: Thu Apr 05 19:39:45 2007

Subject: FW: IEA Deck March 17th - 31st

Wow... some really shitty results this time. Something wrong?
Received on Fri Sep 14 2007 - 10:55:53 BST

This archive was generated by hypermail 2.2.0 : Sun Sep 16 2007 - 22:19:46 BST