NOTE! An anomaly: the hash -> url mapping ALWAYS adds  trailing ? in the url
even if there were no CGI arguments. This is due to insertion code.

The hash-url.db is hash->url mapping. It is created by the python script when
converting records to text data.

Further workflow and datafiles are as follows:

1. raw
------
The raw binary data as parsed from the input logs.

- lne0, asturies0:   the corresponding newspapers 'main' logs (code 0)
- lne1, asturies1:   logs from the streaming web server (code 1)
- lne2, asturies2:   logs from the streaming helix service (code 2)
- lne01, asturies01 [q_joinwebs]: web server logs sorted and merged in time

2. sessions
-----------
Raw binary data with identified sessions and merged both http request streams.
The data is also sorted in time. 

- lne01, asturies01 [q_sessions]: merged web logs of the corresponding news
papers with identified sessions.

3. filtered
-----------
Sessions are filtered to include only some subset of requests.

- lne0vc [q_http_filter]: includes only valid user 'clicks', i.e. item loads
  (no lower-level elements like pictures), frontpage and status codes of 200
  and 304.

4. lifetimes
------------
Apply q_lifetimes to filtered/lne0vc and output to results/raw/lifetimes.txt
We discard sections (article id == 0) because they have largest lifetime and
introduce heavy bias:

awk -F, '$2>0 { print $3 }' results/raw/lifetimes.txt > results/postproc/lifetimes.txt

5. groups of 3 items
--------------------
Apply q_groups to each week of data. In all cases, the groups are calculated
with resolution of 1 hour, and the frontpage (IDs: -1,-1) is included in each
group.

./scripts/q_groups bindata/filtered/lne0vc 2005-05-16T0:0:0 2005-05-23T0:0:0 1 > results/raw/groups_0516_0522.txt
./scripts/q_groups bindata/filtered/lne0vc 2005-05-23T0:0:0 2005-05-30T0:0:0 1 > results/raw/groups_0523_0529.txt
./scripts/q_groups bindata/filtered/lne0vc 2005-05-30T0:0:0 2005-06-06T0:0:0 1 > results/raw/groups_0530_0605.txt
./scripts/q_groups bindata/filtered/lne0vc 2005-06-06T0:0:0 2005-06-13T0:0:0 1 > results/raw/groups_0606_0612.txt
./scripts/q_groups bindata/filtered/lne0vc 2005-06-13T0:0:0 2005-06-20T0:0:0 1 > results/raw/groups_0613_0619.txt
./scripts/q_groups bindata/filtered/lne0vc 2005-06-20T0:0:0 2005-06-27T0:0:0 1 > results/raw/groups_0620_0626.txt
./scripts/q_groups bindata/filtered/lne0vc 2005-06-27T0:0:0 2005-07-04T0:0:0 1 > results/raw/groups_0627_0703.txt
./scripts/q_groups bindata/filtered/lne0vc 2005-07-04T0:0:0 2005-07-11T0:0:0 1 > results/raw/groups_0704_0711.txt
./scripts/q_groups bindata/filtered/lne0vc 2005-07-11T0:0:0 2005-07-18T0:0:0 1 > results/raw/groups_0711_0717.txt
./scripts/q_groups bindata/filtered/lne0vc 2005-07-18T0:0:0 2005-07-25T0:0:0 1 > results/raw/groups_0717_0724.txt

Postprocess the data:
for i in groups_0*; do python2.4 ../../scripts/post_groups.py $i > ../postproc/$i; done

Apply q_groups to the complete data set with resolution of 1 week and
postprocess the data:

./scripts/q_groups bindata/filtered/lne0vc 2005-05-16T0:0:0 2005-07-25T0:0:0 168 > results/raw/groups_all_byweek.txt
python2.4 scripts/post_groups.py results/raw/groups_all_byweek.txt > results/postproc/groups_all_byweek.txt

Don't include groups which had frontpage in it. 
grep -v 4294967295 results/raw/groups_all_byweek.txt > results/raw/groups_all_byweek_nofp.txt
python2.4 scripts/post_groups.py results/raw/groups_all_byweek_nofp.txt > results/postproc/groups_all_byweek_nofp.txt

This is more relevant analysis as the FP introduces a large bias (i.e. zipf
parameter with frontpage is slightly > 1, w/o frontpage in groups is ~0.65).

6. Popularity of items by week
------------------------------
Calculate popularity of all items (secs, fp, and news) in the whole time span,
with 1 week as the accumulation period.
./scripts/q_popularity bindata/filtered/lne0vc 2005-05-16T0:0:0 2005-07-25T0:0:0 168 > results/raw/popularity_all_byweek.txt

Get popularity only for items that are neither sections nor frontpage. Retain
week number for processig within mathematica.

awk -F, '($3 != 0) && ($2 != 4294967295) { print "{" $1 "," $4 "}" }' results/raw/popularity_all_byweek.txt  > results/postproc/popularity_all_byweek.txt

Calculate popularity of all items (secs, fp, and news) in the whole time span,
with 1 day as the accumulation period.
./scripts/q_popularity bindata/filtered/lne0vc 2005-05-16T0:0:0 2005-07-25T0:0:0 168 > results/raw/popularity_all_byweek.txt

Classify popularity into the frontpage (0), sections (1) and items (2). The
output format are lists in the format { time, kind, count } where kind is
0, 1 or 2.

awk -F, '{class=2} ($3==0) { class=1 } ($2==4294967295) {class=0} { print "{" $1 "," class "," $4 "}" }' results/raw/popularity_all_byhour.txt > results/postproc/popularity_classes_all_byhour.txt 

7. Session analysis
-------------------
Calculate the session durations and sizes for each week.

./scripts/q_session_sizes bindata/filtered/lne0vc 2005-05-16T0:0:0 2005-05-23T0:0:0 > results/raw/session_sizes_0516_0522.txt
./scripts/q_session_sizes bindata/filtered/lne0vc 2005-05-23T0:0:0 2005-05-30T0:0:0 > results/raw/session_sizes_0523_0529.txt
./scripts/q_session_sizes bindata/filtered/lne0vc 2005-05-30T0:0:0 2005-06-06T0:0:0 > results/raw/session_sizes_0530_0605.txt
./scripts/q_session_sizes bindata/filtered/lne0vc 2005-06-06T0:0:0 2005-06-13T0:0:0 > results/raw/session_sizes_0606_0612.txt
./scripts/q_session_sizes bindata/filtered/lne0vc 2005-06-13T0:0:0 2005-06-20T0:0:0 > results/raw/session_sizes_0613_0619.txt
./scripts/q_session_sizes bindata/filtered/lne0vc 2005-06-20T0:0:0 2005-06-27T0:0:0 > results/raw/session_sizes_0620_0626.txt
./scripts/q_session_sizes bindata/filtered/lne0vc 2005-06-27T0:0:0 2005-07-04T0:0:0 > results/raw/session_sizes_0627_0703.txt
./scripts/q_session_sizes bindata/filtered/lne0vc 2005-07-04T0:0:0 2005-07-11T0:0:0 > results/raw/session_sizes_0704_0710.txt
./scripts/q_session_sizes bindata/filtered/lne0vc 2005-07-11T0:0:0 2005-07-18T0:0:0 > results/raw/session_sizes_0711_0717.txt
./scripts/q_session_sizes bindata/filtered/lne0vc 2005-07-18T0:0:0 2005-07-25T0:0:0 > results/raw/session_sizes_0718_0724.txt

The otuput of the program is a file in the format:
start_time - the start period
for each session, a line in the format:
sess_id, start_time, last_time

Postprocess the data:
for i in session_sizes_0*; do python2.4 ../../scripts/post_session_sizes.py $i > ../postproc/$i; done

Calculate session durations:
./scripts/q_session_times bindata/filtered/lne0vc 2005-05-16T0:0:0 2005-05-23T0:0:0 > results/raw/session_times_0516_0522.txt
./scripts/q_session_times bindata/filtered/lne0vc 2005-05-23T0:0:0 2005-05-30T0:0:0 > results/raw/session_times_0523_0529.txt
./scripts/q_session_times bindata/filtered/lne0vc 2005-05-30T0:0:0 2005-06-06T0:0:0 > results/raw/session_times_0530_0605.txt
./scripts/q_session_times bindata/filtered/lne0vc 2005-06-06T0:0:0 2005-06-13T0:0:0 > results/raw/session_times_0606_0612.txt
./scripts/q_session_times bindata/filtered/lne0vc 2005-06-13T0:0:0 2005-06-20T0:0:0 > results/raw/session_times_0613_0619.txt
./scripts/q_session_times bindata/filtered/lne0vc 2005-06-20T0:0:0 2005-06-27T0:0:0 > results/raw/session_times_0620_0626.txt

8. Working set for the continuous media
---------------------------------------
Calculate traffic for the streaming server.
python2.4 scripts/q_sws.py rawdata/melandru-lne-20050516_20050724.log 1116194400 > results/postproc/working_set_streaming.txt

