Re: [GT] Opinions on date conversion sought

["Robert A. Schmied" <ras AT acm.org>]

João Costa wrote:
> I think i kinda see the problem Thomas raised in the example.
>
> In mapping weekly into monthly timeframes, you get inconsistent
> results, because data from a weekly period overlaps two different
> monthly periods, so you're discarding the last weekly period for the
> current month calculation.
>
> Also, that bit where you say:
>
> > date computed for week 2008-27 is considered to be part of June (2008/06),
> > but some of the information required for 2008-27 (the days in July that are needed to
> > get the closing price for 2008-27) have not even occurred then
>
>
> This can happen in any timeframe, but the problem here is that a
> period which has not finished yet is being calculated.
> The same thing would happen if for eg: converting from daily to weekly
> on a Wednesday, or from minute to hour at 30minutes past.
>
> In this case I think it's not a problem of the conversion routines,
> it's that we're calculating a period which is still unfinished.
>
> And following Robert's tip, some examples can help clarify things.
>
>
>
> On Wed, Jul 9, 2008 at 8:52 PM, Robert A. Schmied <ras
AT
acm.org> wrote:
> > Weigert, Thomas wrote:
> >
> > > João,
> > >
> > > I ran into this situation in the following way: I had been doing some
> > > performance evaluation of indicators using timeframe 'day'. As these
> > > took a long time to run, I used timeframe 'week' but got some weird
> > > results. (I produce summary tables for backtesting which shows the
> > > performance of a portfolio on a monthly basis over the years
> > > invested.)
> >
> > aloha thomas
> >
> > since you didn't share the command line directives you used we cannot
> > independently empirically evaluate your observations and your conclusions.
> >
> > but i too have experienced issues when converting dates, meaning year
> > month day to year week-of-year. however, this only ever occurred
> > when i, not the gt system, did the conversion. by that i mean i mentally
> > converted 30 june 2008 (or 31 dec 2005) to week of year and usually got
> > it different from what gt would get.
> >
> > that said i have yet to find conversion inconsistencies in the internal
> > date conversions that gt had implemented via GT::DateTime and Date::Calc.
> > it seems consistent in it's handling of these year end date oddities and
> > how one counts weeks of the year.
> > see date and calendar standards for more. the issue isn't simple, there
> > are multiple was to account for the extra days in a year.
> >
> > my solution was to use dates *always* formatted year-mon-day and then if
> > the time frame was set to week or year have the code do the conversion.
> >
> > my perl implementation to do that was in a version of scan.pl posted a
> > while back (and may still be mostly in the trunk version). the key
> > elements: validate user date inputs and then convert them according
> > to the timeframe setting. i relied on Date::Manip for the former and
> > GT::DateTime (and Date::Calc) for the latter to ensure date consistency.
> >
> > the other gt scripts would benefit from similar change.
> >
> > by allowing multiple date string formats that are intended to align with
> > a timeframe there is no way that i can envision to ensure the user's
> > notion of week 53, 0 or 1 is consistent with that of gts'. best way to
> > handle it (best meaning least risk, change, work, ...) is to encourage
> > users to use yyyy-mm-dd and do the time frame conversions internally.
> > do this throughly, with good error feedback and users will, well, use.
> >
> >
> > just my few cents on dates yet again ...
> >
> > ras
> >
> >
> >
> >
> > > Tracking the difference in behavior down I found the issue mentioned
> > > earlier, namely that when mapping a weekly analysis into months we
> > > end up with situations that, e.g., date computed for week 2008-27 is
> > > considered to be part of June (2008/06), but some of the information
> > > required for 2008-27 (the days in July that are needed to get the
> > > closing price for 2008-27) have not even occurred then.
> > >
> > > In such situations it will be better to consider the week to belong
> > > to July, as then we deal only with realistic situations.
> > >
> > > Whether this is an issue or not depends on the type of analysis you
> > > are doing, and the types of summary tables you produce, if any.
> > >
> > > I am not sure whether it will always be better to consider dates
> > > based on ending dates in conversion, or whether this is only better
> > > in the situation I ran into.
> > >
> > > In the meantime, I have implemented DateTime routines that allow
> > > conversions between timeframes based on end dates, so that users have
> > > the option to rely on those when needed...
> > >
> > > Cheers, Th.
> > >
> > >
> > >
> > > > -----Original Message----- From: João Costa
> > > > [mailto:joaocosta
AT
zonalivre.org] Sent: Tuesday, July 08, 2008 7:42
> > > > PM To: devel
AT
geniustrader.org Subject: Re: [GT] Opinions on date
> > > > conversion sought
> > > >
> > > > For some reason I stopped receiving messages from the mailing list
> > > > for the last 3 days, sorry that i keep breaking the topic index
> > > > that keeps messages neatly organized by topic.
> > > >
> > > > On the particular issue of date conversion as you exposed it, just some
> > > > questions:
> > > >
> > > > - Is there any particular disadvantage on the way GT currently
> > > > handles this ? - How did you come across this ? What's the
> > > > motivation to change this ?
> > > >
> > > >
> > > > From what I could understand from your message, I get the
> > > > impression that doing this in different ways will yield different
> > > > results, but it seems neither way is better/worse than the other,
> > > > it's just different. Is this correct ?


without more specifics from thomas i still think gts' internal conversions
are 'consistent', they might not be logical from a human conception of date
conversion, but they seem consistent to me.

i don't expect gt to work when i specify 'tomorrow' or 'next friday' in day
timeframe, in weekly timeframe specifying a day in any week is essentially
specifying that entire week. similarly specifying a month means all days
in that month (even if the month is not yet over), year means the entire year.

on what criteria is the decision to back up to the end of the prior week
(month, year) being made?




using the attached hack you can experiment with date string inputs, it
uses Date::Manip if installed, but only to ensure the date string is
yyyy-mm-dd, after that it uses GT:DateTime for the timeframe conversion.
enter date strings in gt timeframe other than day at your own risk.

31dec07 is in first week of 2008
%     gt_date_conv.pl --ti week   '31dec07' 
pre timeframe adjust:
date:   2007-12-31 00:00:00
start:  
end:    

post timeframe adjust:
date:   2008-01
start:  
end:    


30jun08 is in 27th week of 2008

%     gt_date_conv.pl --ti week   30jun08

pre timeframe adjust:
date:   2008-09-30 00:00:00
start:  
end:    

post timeframe adjust:
date:   2008-40
start:  
end:    


%     gt_date_conv.pl --ti month  30jun08

pre timeframe adjust:
date:   2008-06-30 00:00:00
start:  
end:    

post timeframe adjust:
date:   2008/06
start:  
end:    

%      gt_date_conv.pl --ti month  27jun08

pre timeframe adjust:
date:   2008-06-27 00:00:00
start:  
end:    

post timeframe adjust:
date:   2008/06
start:  
end:    
#!/usr/local/bin/perl -w

#!/usr/bin/perl -w

# Copyright 2008 ras based on and for gt Copyright 2000-2003 Raphaël Hertzog
# This file is distributed under the terms of the General Public License
# version 2 or (at your option) any later version.

#
# $Id$

use lib '..';

use strict;
use vars qw($db);

#use GT::Prices;
use GT::Calculator;
#use GT::List;
#use GT::Eval;
use GT::Conf;
use GT::Tools qw(:conf :generic :timeframe);
use GT::DateTime;
use Getopt::Long;
use Pod::Usage;

use Date::Calc qw( Date_to_Days );

GT::Conf::load();

( my $prog_name = $0 ) =~ s
AT
^.*/@@;    # lets identify ourself

# Manage options
my ($full, $verbose, $timeframe, $start, $end, ) = 
   (0,     0,        'day',      '',     '',   );

my @options;

GetOptions( 'full!'		=> \$full,
            'verbose+'		=> \$verbose,
            'timeframe=s'	=> \$timeframe,
            "start=s"		=> \$start,
            "end=s"		=> \$end,
            "option=s"		=> \
AT
options,
           );

if ( $verbose > 0 ) {
  print STDERR "$prog_name: verbose=$verbose\n";
}

# process all options from gt options file
foreach (@options) {
    my ($key, $value) = split (/=/, $_);
    GT::Conf::set($key, $value);
}


# get date string from command line
my $date = shift;


# assumptions: date is the day of interest
#              start and end dates define time span under analysis
#              date is expected to be within that time span
#              timeframe is the time period chunk size
#              start and end dates must match the selected timeframe
#
#
# datetime formats permitted
# yyyy-mm-dd with or without leading zeros
# yyyymmdd with required leading zeros
# <date> hh:mm:ss with required separator and leading zeros
if ( ! $date ) {
  print STDERR "$prog_name: error: require date parameter\n\n";
  print STDERR "date formats are YYYY-MM-DD with or without leading zeros\n";
  print STDERR "                 YYYYMMDD leading zeros required\n\n";
  print STDERR "time formats for sub day timeframes is:\n";
  print STDERR "                 <date> HH:MM:SS\n\n";
  print STDERR "explicit timeframe required if time included in date\n\n";
  usage();
  exit 1;
}

my $tf;
my $time;
my $err_msg;

my $in_date = $date;
my $in_start = $start if ( $start );
my $in_end = $end if ( $end );

if ( $timeframe ) {
  $tf = GT::DateTime::name_to_timeframe($timeframe);
} else {
  # assume default is $DAY timeframe
  $tf = $DAY;
}

my ( $d_yr, $d_mn, $d_dy, $d_tm );
if ( ! parse_date_str( \$date, \$err_msg ) ) {
  die "$prog_name: error: $err_msg\n";
} else {
  ( $d_yr, $d_mn, $d_dy, $d_tm ) = split /[- ]/, $date;
}

my ( $s_yr, $s_mn, $s_dy, $s_tm );
if ( $start ) {
  if ( ! parse_date_str( \$start, \$err_msg ) ) {
    die "$prog_name: error: $err_msg\n";
  } else {
    ( $s_yr, $s_mn, $s_dy, $s_tm ) = split /[- ]/, $start;
  }
}

my ( $e_yr, $e_mn, $e_dy, $e_tm );
if ( $end ) {
  if ( ! parse_date_str( \$end, \$err_msg ) ) {
    die "$prog_name: error: $err_msg\n";
  } else {
    ( $e_yr, $e_mn, $e_dy, $e_tm ) = split /[- ]/, $end;
  }
}

if ( $start && $end ) {
  # $start must be prior to $end
  if (Date_to_Days($s_yr, $s_mn, $s_dy) >=
      Date_to_Days($e_yr, $e_mn, $e_dy)) {
    warn "$prog_name: --start date must be prior to --end date ($start before $end)\n";
  }
}
  
if ( $date && $end ) {
  # $date must be $end or before
  if (Date_to_Days($d_yr, $d_mn, $d_dy) >
      Date_to_Days($e_yr, $e_mn, $e_dy)) {
    warn "$prog_name: date must be prior to or equal --end date ($date before $end)\n";
  }
}
  
if ( $date && $start ) {
  # $start must be prior to $date
  if (Date_to_Days($s_yr, $s_mn, $s_dy) >=
      Date_to_Days($d_yr, $d_mn, $d_dy)) {
    warn "$prog_name: --start must be prior to date ($start before $date)\n";
  }
}

  print STDERR "\npre timeframe adjust:\n";
  print STDERR "date:\t$date\n";
  print STDERR "start:\t$start\n";
  print STDERR "end:\t$end\n";

# timeframe relative date conversions
if ( $start && $tf != $DAY ) {
  $start = GT::DateTime::convert_date($start, $DAY, $tf);
}

if ( $end && $tf != $DAY ) {
  $end = GT::DateTime::convert_date($end, $DAY, $tf);
}

if ( $tf != $DAY && $tf > $DAY ) {
  $date = GT::DateTime::convert_date($date, $DAY, $tf);
}

  print STDERR "\npost timeframe adjust:\n";
  print STDERR "date:\t$date\n";
  print STDERR "start:\t$start\n";
  print STDERR "end:\t$end\n\n";





sub usage {
  print STDERR "$prog_name [ options ] date\n";
  print STDERR "\n";
  print STDERR "where standard date format is YYYY-MM-DD\n";
  print STDERR "\n";
  print STDERR "      date can include optional time: <date>' HH:MM:SS'\n";
  if ( eval { require Date::Manip } ) {
    print STDERR "\n  ah! since you have Date::Manip available date strings can also be specified\n";
    print STDERR "  in any format that Date::Manip can parse. common useful strings include:\n";
    print STDERR "  'today', 'yesterday', 'last friday', '6 months ago', '1st of last month'\n";
    print STDERR "  are all simple examples that make date entry much more human-date relative\n";
    print STDERR "  \"perldoc -t Date::Manip\" for the gory details on date string parsing\n";
  }  
  print STDERR "\n";
}