Dev Status Meeting - 7 Sept 2007

From: Ben Grodsky <>
Date: Fri, 7 Sep 2007 14:51:03 -0700



* http downloading done on app.

        * Currently just deletes partial file and starts over

* applet console done for NY AG

        * working on http downloading there

        * should be deployable Monday

        * working on resume for those

* have some Proxy fixes for QA


* DC for ed2k working for demand (nothing yet for supply)

        * on windows machines

        * only collecting 1/10 of demand, b/c so much

        * ~10MM rows from 1 week of collection

        * table keyed on IP of incoming search, the actual search, and the date (this removes redundancy of equal searches from same IP on multiple machines ???)

        * have to work on multiple project data processing

* linux servers

        * became unstable

        * socket connection issues

        * can't get any new incoming traffic, so they appear dead from outside, but not sure why

* miivi

        * gave Colin RSS tool to input feeds

        * redid filtering system to ???? using PM ???

        * giving Gerald info whether video is part of channel

        * talked to Bobby about channel page

                * top Bobby priority is getting asset for channel video

                * then getting header assets

                * then working on channel page

        * don't wait on Bobby for 1st pass of Channel page, work out with Gerald a hybrid of the media page and search page ????

                * need testers to start using channel functionality

                * look and feel is secondary to functionality

        * new mac version on hold from Ty b/c of Ares crisis


* got lots of proxy IPs from BT.

        * Have talked with Ebert

        * wireshark what a proxy does.

        * Talk with Sergio about classes that he's written

                * usually gets are different

                * the "forurl" is the same ???

* have interface requests for Dylan (possibly Sheetal)

* gearing up for TV season.

        * Night shift starting soon

        * AP tweaks (for Jed) about using previous week's meta data and just changing episode #

* Hann

        * scheduling appt with Randy, Jay

        * working on getting him gnutella data

                * can provide August data

                * Danny and Sujay to coordinate

                * Danny to delete old Hann data from QA ftp

        * trying to get special version of preliminary report from him marked Confidential everywhere. Otherwise, just have to tell everyone that touches the report that it's confidential.


* miivi

        * adding ajax to media page

        * battled quicktime embed, but no dice so far.

                * Start with samples that you know work.

                * Starting from absolute scratch

        * Dylan's giving media page stuff

* BT searcher

        * made changes for meganova and a few other sites

* BT processor

        * having passive/possible db issues

        * though there have been no changes, all of a sudden it's not working quite right.

        * Removing monitoring projects


* errors in saving project name (incorrect length)

* most hubs not accepting connections, saying it's already full

* have increased # of allowable connections

* connect --> handshake --> local node info

* hub returns ping --> process packet for udp search

* after sending it, hub returns something but then the socket closes unexpectedly

        * now connecting with actual shareaza client and checking byte for byte what's going on in wireshark

* jeff's code became unstable

        * controller being restarted takes a long time now (took ~2 hrs)

        * restarting leaves time consuming too

* keeping track of failed tracks from QA so can check for those tracks


* problem with eMule downloader resolved, was offline for short period of time.

* inputting UMG hashes had issue; got resolved.

* Was manually ftping for UMG; now that's automatically dumping to them daily.


* inflation should be able to take higher load now

* got 250 IPs on 5 machines: those machines handling load well; <1% cpu load; ~250mb ram load; minimal bw use

* haven't tested actual number appearance on website

        * think of some way for QA to test this

        * also been concentrating on BT website for Benny (legacy Jeff site)

                * QA could use this new BT website to test inflation effect

                * organized version of scraped torrents db


* working on back-end editing artists, tracks, etc.

        * basically done

        * prettifying for PM team

* fixed geographic score to correlate better with daily score

* tracking down automated script bug

        * currently running just once each day

        * when run script manually (outside chron job) it works fine

* may be problem with track reconciliation from PM

        * i.e., projects turned on for MI conflicting with info from PM

        * some issues may cause changes to revert to previous version

        * talk with Jay if need help

* a couple machines were behaving very oddly - collecting 10x as much data as others right next to them, causing increased cpu use

        * this occurring days on end doesn't mean much

        * but if this occurs even after system restarts, then there's something odd going on - 2 really good nodes (1 from us and 1 from another anti-p2p company, a datacenter dude fooling around with a lot of bandwidth on gnutella, etc.).

                * if you're caching IPs, this


* fixing small slsk bugs

* servers went down over long weekend; came in Monday to fix

* Sujay is probably able to provide you a replacement network for your slsk

* working on 137 now. Starting to look at 156??


* all websites added

* btjunkie being tested for ~1 day.

        * Working fine in testing mode

        * not working well in release mode

* adding new categories to poster (should be done early next week -- ~Tuesday)

* want to add logs about which site posts succeeded

* all these changes should be present in next version


* a lot of problems over long weekend

* stability increased during week

* lose Internet connection after ~48 hours

        * may be worthwhile just to kill the process once daily to avoid that state.

        * You still want to find the problem, but may be able to keep up productivity just by using a Trapper Keeper-The Keeper type set-up to make sure the program relaunches ok

* rate of download has gone down (as expected), because caught up now with things that can download b/c so many sources; but the new stuff has so few sources that it's harder to download those

* make sure that the # of hashes you're seeing and the # downloaded each day makes sense: if #s drop down to 0 or nearly 0, that may indicate a problem. Drop-off is expected, but an extreme drop-off would raise a red-flag


* NY atty gen

        * optimized ??? processing

        * changed schema

        * reprocessed data

* gnutella protection

        * had some fails this week

        * making controlling changes so don't have to recompile code changes in same way

* kad should be done

* reformatted computer, but forgot to back-up some Dbs


* changed dir structure to match hash and filenames are now the original p2p filename

* now downloading from NY IPs only (Serg's change)

* out of 5700 files, ~100 downloaded

* ready to put on NY AG system for additional testing

* logging

        * keep the 200 ok accept to the http header

        * store date, time, ip, http header (including range of data - which pieces)

* h264 flash

        * clearly progressive, not streaming

        * haven't seen players that work progressively

        * may be similar to wii flash player (dylan to send link)

        * fla is source (talk to Sujay about how to open that file and make changes to it)

                * app will be acting as local web server

                * flv has progressive download and it supports h264, but can you do both?

                        * If yes, then we'll talk to Stephan
Received on Fri Sep 14 2007 - 10:55:50 BST

This archive was generated by hypermail 2.2.0 : Sun Sep 16 2007 - 22:19:45 BST