[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [GT] Opinions on date conversion sought

implemented DateTime routines that allow

conversions between timeframes based on end datesthomas



we are back at the beginning again. weeks don't align with months,
that's a problem business data processing has struggled with for years.

so how have you gathered the extra days and or missing days info into
a weekly data set without introducing other data anomalies?

only truly advanced users would ever think of performing step 2 below.
how is it done exactly?


aloha


ras


Weigert, Thomas wrote:

João,

Here is a simple example:

1. We compute the monthly close of BRPIX, for example (note that in the data set used below, the last day was 2008-07-03):

./display_indicator.pl --start=2007-10-1 timeframe=month I:Prices BRPIX CLOSE
Prices[CLOSE]       [2007/10] = 24.1000
Prices[CLOSE]       [2007/11] = 25.2000
Prices[CLOSE]       [2007/12] = 25.5200
Prices[CLOSE]       [2008/01] = 27.2300
Prices[CLOSE]       [2008/02] = 28.1000
Prices[CLOSE]       [2008/03] = 28.2000
Prices[CLOSE]       [2008/04] = 26.8500
Prices[CLOSE]       [2008/05] = 26.5500
Prices[CLOSE]       [2008/06] = 28.9500
Prices[CLOSE]       [2008/07] = 29.3400

2. Now create the BRPIX data set in weekly timeframe, and compute the monthly close again:

./display_indicator.pl --start=2007-10-1 --timeframe=month I:Prices BRPIXweekly CLOSE
Prices[CLOSE]       [2007/10] = 24.7200
Prices[CLOSE]       [2007/11] = 25.2000
Prices[CLOSE]       [2007/12] = 26.5500
Prices[CLOSE]       [2008/01] = 26.8500
Prices[CLOSE]       [2008/02] = 28.1000
Prices[CLOSE]       [2008/03] = 27.1600
Prices[CLOSE]       [2008/04] = 26.3000
Prices[CLOSE]       [2008/05] = 26.5500
Prices[CLOSE]       [2008/06] = 29.3400


Notice the difference? Most months are off, due to the difference in which month days are being counted as belonging. For example, the close for Jun in the first set does not include 3 Jul, which is the close of Jun in the weekly dataset.
Of course, that the data does not agree is unavoidable. But what I think is not good is that the second set includes the CLOSE of days not in this month. Note that all the OPEN are in the correct month, of course. 
However, most analysis is done on CLOSE, so I think it would be better to ensure that the CLOSE is in the correct month.

Th.

I attach the weekly BRPIX for your entertainment...



-----Original Message-----
From: João Costa [mailto:joaocosta

AT
zonalivre.org]

Sent: Wednesday, July 09, 2008 3:24 PM
To: devel

AT
geniustrader.org

Subject: Re: [GT] Opinions on date conversion sought

I think i kinda see the problem Thomas raised in the example.

In mapping weekly into monthly timeframes, you get inconsistent
results, because data from a weekly period overlaps two different
monthly periods, so you're discarding the last weekly period for the
current month calculation.

Also, that bit where you say:


date computed for week 2008-27 is considered to be part of June


(2008/06),


but some of the information required for 2008-27 (the days in July


that are needed to


get the closing price for 2008-27) have not even occurred then


This can happen in any timeframe, but the problem here is that a
period which has not finished yet is being calculated.
The same thing would happen if for eg: converting from daily to weekly
on a Wednesday, or from minute to hour at 30minutes past.

In this case I think it's not a problem of the conversion routines,
it's that we're calculating a period which is still unfinished.

And following Robert's tip, some examples can help clarify things.



On Wed, Jul 9, 2008 at 8:52 PM, Robert A. Schmied <ras

AT
acm.org> wrote:



Weigert, Thomas wrote:


João,

I ran into this situation in the following way: I had been doing


some


performance evaluation of indicators using timeframe 'day'. As these
took a long time to run, I used timeframe 'week' but got some weird
results. (I produce summary tables for backtesting which shows the
performance of a portfolio on a monthly basis over the years
invested.)



aloha thomas

since you didn't share the command line directives you used we cannot
independently empirically evaluate your observations and your


conclusions.


but i too have experienced issues when converting dates, meaning year
month day to year week-of-year. however, this only ever occurred
when i, not the gt system, did the conversion. by that i mean i


mentally


converted 30 june 2008 (or 31 dec 2005) to week of year and usually


got


it different from what gt would get.

that said i have yet to find conversion inconsistencies in the


internal


date conversions that gt had implemented via GT::DateTime and


Date::Calc.


it seems consistent in it's handling of these year end date oddities


and


how one counts weeks of the year.
see date and calendar standards for more. the issue isn't simple,


there


are multiple was to account for the extra days in a year.

my solution was to use dates *always* formatted year-mon-day and then


if


the time frame was set to week or year have the code do the


conversion.


my perl implementation to do that was in a version of scan.pl posted


a


while back (and may still be mostly in the trunk version). the key
elements: validate user date inputs and then convert them according
to the timeframe setting. i relied on Date::Manip for the former and
GT::DateTime (and Date::Calc) for the latter to ensure date


consistency.


the other gt scripts would benefit from similar change.

by allowing multiple date string formats that are intended to align


with


a timeframe there is no way that i can envision to ensure the user's
notion of week 53, 0 or 1 is consistent with that of gts'. best way


to


handle it (best meaning least risk, change, work, ...) is to


encourage


users to use yyyy-mm-dd and do the time frame conversions internally.
do this throughly, with good error feedback and users will, well,


use.



just my few cents on dates yet again ...

ras





Tracking the difference in behavior down I found the issue mentioned
earlier, namely that when mapping a weekly analysis into months we
end up with situations that, e.g., date computed for week 2008-27 is
considered to be part of June (2008/06), but some of the information
required for 2008-27 (the days in July that are needed to get the
closing price for 2008-27) have not even occurred then.

In such situations it will be better to consider the week to belong
to July, as then we deal only with realistic situations.

Whether this is an issue or not depends on the type of analysis you
are doing, and the types of summary tables you produce, if any.

I am not sure whether it will always be better to consider dates
based on ending dates in conversion, or whether this is only better
in the situation I ran into.

In the meantime, I have implemented DateTime routines that allow
conversions between timeframes based on end dates, so that users


have


the option to rely on those when needed...

Cheers, Th.




-----Original Message----- From: João Costa
[mailto:joaocosta

AT
zonalivre.org] Sent: Tuesday, July 08, 2008 7:42

PM To: devel

AT
geniustrader.org Subject: Re: [GT] Opinions on date

conversion sought

For some reason I stopped receiving messages from the mailing list
for the last 3 days, sorry that i keep breaking the topic index
that keeps messages neatly organized by topic.

On the particular issue of date conversion as you exposed it, just


some


questions:

- Is there any particular disadvantage on the way GT currently
handles this ? - How did you come across this ? What's the
motivation to change this ?


From what I could understand from your message, I get the
impression that doing this in different ways will yield different
results, but it seems neither way is better/worse than the other,
it's just different. Is this correct ?