[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [GT] Re: gt-d] Trouble with backtest.pl and beancounter

Weigert, Thomas wrote:

Here is the blow-by-blow of this problem, in case you are interested
plus possible resolutions....


In GT::Eval::create_db_object, the current db connection is reused if
the $db variable is defined (the assumption the code is making here is
that if the variable is defined, then the connection to the data base
exists).


When we disconnect from the data base, this variable is not undefined,
and thus, the attempt to reuse above fails, leaving us to access a
disconnected data base.


There are two possible ways to deal with this issue:


1.       Move the responsibility of connecting/disconnecting from the
data base to each script (as it was before the factoring).


this might be the preferred way. all script apps, 'supported and modified',
'official but forgotten and unmodified' and any home-made apps, utils, etc
are now as they were.



2.       Fix the manner in which the db reused by ensuring that we do
not attempt to reuse a disconnected data base.


this 'sounds' appealing but there's probably a but, there always is.




(1) has the advantage of giving the user more control over the data base
connects. E.g., in backtest.pl, we would not have to reconnect to the
database in order to perform the analysis report (Report.pm and
Indicators/Prices.pm are the only places where the reuse of the db
connection would have any impact).



what is the cost of that reconnect? if it is appreciable then this
is the preferred solution; no changes, plus better performance.


(2) has the advantage of hiding the db access completely from the script
writer.



i'm looking at all the stuff that was crafted based on the past method and
don't see a big payoff for the change (beyond script writer convenience
which is essentially nil since the scripts have already been written)


I have traced data base connections in the scripts through example runs.
It turns out that the only occasions where we have an additional db
connection due to the earlier disconnect under (2) are:

backtest_multi.pl: One connect per market instead of one single connect.


although backtest_multi.pl might be little used, this connect per market
seems to be technically less than optimal unless there's a reason for it.



backtest.pl: One additional connect for the report.


I also discovered that there is an optimization possible in
Indicator/Prices which avoids reconnect to the data base, but that
applies to either scenario.



what is the cost to reconnect? what is the downside if the script (app)
dies without doing a disconnect?


Any preferences for (1) or (2)?



while i have yet to see any performance improvement from the change
to (2) from (1) i see little reason reverting back to (1) so long as
the old methods are supported and continue to work indefinitely.

however, if the final version of (2) results in removal/non-operation of
these methods:
  my $db = create_db_object();

followed by

  my $q = $db->get_prices($code);
  my $calc = GT::Calculator->new($q);
  $calc->set_code($code);

then i prefer (1) over (2).

ras



Thanks, Th.


From: Weigert, Thomas [mailto:weigert

AT
mst.edu] 
Sent: Sunday, May 04, 2008 1:36 PM
To: devel

AT
geniustrader.org

Subject: RE: [GT] Re: gt-d] Trouble with backtest.pl and beancounter


I have installed beancounter and PG on my system to look at this
problem. 
It turns out that throughout GT there are locations where a data base is
reused via a shared "our" variable. While it appears when one looks at
the code that the $db reference is assigned to a "my" variable, in fact
the content of that variable is retrieved from the shared reference.
Therefore, the disconnect from the data base must be done only once, as
there are situations where the assumption is being made that the data
base has already been connected to. While this is not very safe, and not
documented, it does imply that we must make the assumption that the data
base should be connected to in the script and disconnected from at the
end of the script.


Commenting the disconnect out won't do much harm, but the better way
will be to migrate the disconnect back into the scripts.


On cygwin (the environment I am running, I am getting problems with fork
and db access, so there are issues with running scan.pl with
beancounter. Thus I have not been able to look at scan.pl, due to that
problem. 
Th.


From: Nicholas Kuechler [mailto:nkuechler

AT
gmail.com] 
Sent: Tuesday, April 29, 2008 2:45 PM
To: devel

AT
geniustrader.org

Subject: Re: [GT] Re: gt-d] Trouble with backtest.pl and beancounter


With Roberts help, we found the DB disconnect in GT/Tools.pm appears to
be causing this problem.

I've commented it out for now:

  #$db->disconnect;

I'm not sure commenting it out complete is "ideal" but I'm not sure
where else to place this line of code to produce a working solution.