[GT] IB, tree; was: Opentick

[R P Herrold <herrold AT owlriver.com>]

On Thu, 10 Jul 2008, João Costa wrote:

> I tried building it yesterday but it wasn't a svn snapshot, rather one
> of the releases.

> It also failed to build but it was a minor issue in one of the header
> files, a function with duplicate argument names.

I get a different error (a missing library target, rather than 
something trivial to tweak) with tree-1.1.0.2.19_20080312.tar.gz
which I have filed upstream:

http://sourceforge.net/tracker/index.php?func=detail&aid=2015270&group_id=3809&atid=103809

> Have you used it ?  What's your opinion about it ?

I was hopeful that the 'tree' project's goal:

> The TREE Data Server captures real-time financial data from 
> one or several datafeed services (e.g. IB TWS API), archives 
> data in a historical database, and makes both live and 
> archived data available to client applications.

seemed 'tailor made' to extend what were locally cobbled 
together management of datasets when I first looked at the 
description.  I was not at all interested in the IB or TWS 
data connector matters (but as as to the other connectors 
which have been implemented), as we have had those working 
against IB/TWS for quite a while. (over 5 years for Yahoo 
Finance EOD data; IB intra-day, over two years for tickstream, 
and a year and a half for history retrieval, in the shim)

I had sent a private email to the 'tree' project's author 
about some questions I had on symbology which reading 
'ib-notes' in the release had raised, and had not heard back 
....

So near as I make out, a daemon process, called' parser' runs, 
per settings in a config file, and accepts (or notes the 
absence) of data feeds from varioious market data sources; 
the 'parser' then handles insertions into the data stores 
outlined in 'data-storage'.  various routes for maintenance 
(unloading, reloading datasets) exist.

In part, 'data-storage' notes:

> We will store historical data as individual ticks, 5- or 
> 15-minute bars,and daily bars.  This will allow standard bar 
> widths to be generated quickly, and will make it easier to 
> store long historical data series for slow-starting signal 
> processing schemes.

> Intraday bars may be either 5 or 15 minutes wide.

... to some extent, IB's offerings down to 'per second' 
history retrieval suggest that the capabilities of 'tree' may 
need extension (but than very high frequency trading 
strategies may simply be outside of what hte program author is 
interested in)

The 'database' store used appears to be flatfiles of a defined 
format, which is perfectly reasonable ... the *nix filesystem 
is, after all, a 'database' and with well named files (the 
conventions the author uses are outlines), probably 'faster' 
than any SQLish approach.

again quoting:

$TREE/data/info/                        Root of parameter file storage area
$TREE/data/info/XX/INST.info            Instrument parameter file for INST
$TREE/data/info/XX/INST.root            Future-root parameter file for INST
$TREE/data/info/XX/INST.opt             Option-root parameter file for INST
$TREE/data/FEED                 Root of data storage area for datafeed FEED
$TREE/data/FEED/XX/INST/        Data directory for instrument INST (eg, SPU4)
$TREE/data/FEED/XX/INST/YYYY.day        Daily-bar data file for year YYYY
$TREE/data/FEED/XX/INST/YYYYMM.5m       5-minute-bar data file for month MM
$TREE/data/FEED/XX/INST/YYYYMM.15m      15-minute-bar data file for month MM
$TREE/data/FEED/XX/INST/YYYYMM.tick     Tick data file for month MM

Client 'readers' then can do RO accesses at will and without a 
need to consider locking issues.

I speak here from a static file analysis.  Unfortunately as I 
have not gotten a clean compile, and the author has been 
non-responsive, I lost interest, and No, I have not been able 
to get it working in testing.  Understanding the design goals 
and effort is well worth the read, however, and my opinion for 
its potential is high.

I did not immediate locate the backtesting 'reader' scripts 
implied in the documentation, and these may have been 
intentionally elided.

Similar thought underlies the choices we made for the 'shim' 
of course, and we have reached some different choices.

-- Russ herrold