Monday, February 16, 2009

Proof Of Concept 8 of 8b : Historical Quote

The process of retrieving historical data is very much like getting market data. A call to reqHistoricalData() is made with specifications on how far back to request data and how that data is to be seen (also known as bar size settings). A call back function called historicalData() is used to display the data returned. Going into this, I thought it would be a snap to implement, but I encountered some technical difficulties. The problem this time is that Java code works, but JRuby code doesn't - results returned to historicalData() are incorrect.

Here is the Java version. Note that it's pretty straight forward. HistoricalQuote implements EWrapper interface and historicalData() is where the call back happens:

package ibjava;

import ib.client.*;


public class HistoricalQuote implements EWrapper {

public HistoricalQuote() throws InterruptedException {
int i = 0;
EClientSocket m_client = new EClientSocket(this);
m_client.eConnect("127.0.0.1",7496, 0);
System.out.println("Connected.");

// preparing to quote SPY
Contract contract;
contract = new Contract();

contract.m_symbol = "SPY";
contract.m_secType = "STK";
contract.m_expiry = "";
contract.m_right = "";
contract.m_multiplier = "";
contract.m_exchange = "SMART";
contract.m_primaryExch = "";
contract.m_currency = "USD";
contract.m_localSymbol = "";
// submit historical data for SPY.
// look for 3 days worth of trades ending on 02/13/2009,
// grouping results to 1 day at a time
// results will come in through historicalData() methods
m_client.reqHistoricalData(2,
contract,
"20090213 20:00:0", //yyyymmdd hh:mm:ss
"3 D", // duration,
"1 DAY", // barSizeSetting,
"TRADES", // whatToShow,
1, // useRTH,
1) ; // formatDate);
Thread.sleep(4000); // sleep for 4 seconds

m_client.eDisconnect();
System.out.println("Disconnected.");
}

public static void main(String[] args) throws InterruptedException {
System.out.println("Hello");
HistoricalQuote myBot= new HistoricalQuote();
}

// ---------------------------------------------------
// -- methods from EWrapper -- must be declared
// ---------------------------------------------------
public void tickPrice( int tickerId, int field, double price, int canAutoExecute){}
public void tickSize( int tickerId, int field, int size){}
public void tickOptionComputation( int tickerId, int field, double impliedVol, double delta, double modelPrice, double pvDividend){}
public void tickGeneric(int tickerId, int tickType, double value){}
public void tickString(int tickerId, int tickType, String value){}
public void tickEFP(int tickerId, int tickType, double basisPoints, String formattedBasisPoints, double impliedFuture, int holdDays, String futureExpiry, double dividendImpact, double dividendsToExpiry){}
public void orderStatus( int orderId, String status, int filled, int remaining, double avgFillPrice, int permId, int parentId, double lastFillPrice, int clientId, String whyHeld){}
public void openOrder( int orderId, Contract contract, Order order, OrderState orderState){}
public void updateAccountValue(String key, String value, String currency, String accountName){}
public void updatePortfolio(Contract contract, int position, double marketPrice, double marketValue, double averageCost, double unrealizedPNL, double realizedPNL, String accountName){}
public void updateAccountTime(String timeStamp){}
public void nextValidId( int orderId){}
public void contractDetails(int reqId, ContractDetails contractDetails){}
public void bondContractDetails(int reqId, ContractDetails contractDetails){}
public void contractDetailsEnd(int reqId){}
public void execDetails( int orderId, Contract contract, Execution execution){}
public void updateMktDepth( int tickerId, int position, int operation, int side, double price, int size){}
public void updateMktDepthL2( int tickerId, int position, String marketMaker, int operation, int side, double price, int size){}
public void updateNewsBulletin( int msgId, int msgType, String message, String origExchange){}
public void managedAccounts( String accountsList){}
public void receiveFA(int faDataType, String xml){}

public void historicalData(int reqId, String date, double open, double high, double low, double close, int volume, int count, double WAP, boolean hasGaps)
{ System.out.println("Java> id=" + reqId + " date = " + date +
" open=" + open + " high=" + high +
" low=" + low + " close=" + close +
" volume=" + volume + " count=" + count +
" WAP=" + WAP + " hasGaps=" + hasGaps);
}

public void scannerParameters(String xml){}
public void scannerData(int reqId, int rank, ContractDetails contractDetails, String distance, String benchmark, String projection, String legsStr){}
public void scannerDataEnd(int reqId){}
public void realtimeBar(int reqId, long time, double open, double high, double low, double close, long volume, double wap, int count){}
public void currentTime(long time){}
public void fundamentalData(int reqId, String data){}

// ---------------------------------------------------
// -- methods from AnyWrapper -- must be declared
// ---------------------------------------------------

public void connectionClosed() {
System.out.println(" [API.connectionClosed] Closed connection with TWS");
}
public void error(String str) {
System.out.println(" [API.msg1] " + str);
}

public void error(int one, int two, String str) {
System.out.println(" [API.msg2] " + str + " {" + one + ", " + two + "}");
}

public void error(Exception e) {
System.out.println(" [API.msg3] " + e.getMessage());
}

}


Running this produces the following output:

Hello
Server Version:43
TWS Time at connection:20090215 22:48:29 EST
Connected.
[API.msg2] Market data farm connection is OK:usopt {-1, 2104}
[API.msg2] Market data farm connection is OK:usfarm {-1, 2104}
[API.msg2] HMDS data farm connection is OK:ushmds2a {-1, 2106}
Java> id=2 date = 20090211 open=83.44 high=84.05 low=82.4 close=83.53 volume=2950030 count=855129 WAP=83.4 hasGaps=false
Java> id=2 date = 20090212 open=82.17 high=84.1 low=81.05 close=83.98 volume=4193652 count=1201155 WAP=82.25 hasGaps=false
Java> id=2 date = 20090213 open=83.55 high=84.24 low=82.35 close=82.46 volume=2681140 count=795253 WAP=83.44 hasGaps=false
Java> id=2 date = finished-20090210 20:00:00-20090213 20:00:00 open=-1.0 high=-1.0 low=-1.0 close=-1.0 volume=-1 count=-1 WAP=-1.0 hasGaps=false
Disconnected.


Everything is ok at this point. I checked open, high, low, close for each day and the data is accurate.
Implementing this same code in JRuby is as follows:


require 'java'
require 'TwsApi.jar'

include_class 'ib.client.EClientSocket'
include_class 'ib.client.Contract'
include_class 'ib.client.AnyWrapper'
include_class 'ib.client.EWrapper'


class TestHistData
include EWrapper #note "class TestMktData < EWrapper" syntax is no longer supported in JRuby 1.0

@mySocket

def initialize
@mySocket = EClientSocket.new(self)
end

def eConnect
@mySocket.eConnect("127.0.0.1", 7496, 0)
end

def requestHistStockQuote(id, ticker)
# --------------------------------
myContract = Contract.new
myContract.m_symbol = ticker
myContract.m_secType = 'STK'
myContract.m_exchange = 'SMART'
myContract.m_currency = 'USD'
request_id = id
ticklist = ''
# submit historical data for ticker
# look for 3 days worth of trades ending on 02/13/2009,
# grouping results to 1 day at a time
# results will come in through historicalData() methods
@mySocket.reqHistoricalData(request_id,
myContract,
'20090213 20:00:0', #yyyymmdd hh:mm:ss
'3 D', # duration,
'1 DAY', # barSizeSetting,
'TRADES', # whatToShow,
1, # useRTH,
1) ; # formatDate);
end

def eDisconnect
@mySocket.eDisconnect
end


#///////////////////////////////////////////////////////////////////////
#// Interface methods
#///////////////////////////////////////////////////////////////////////

def tickPrice( ticker_id, field, price, can_auto_execute)
end

def tickSize( ticker_id, field, size)
end

def tickOptionComputation( ticker_id, field, implied_vol,
delta, model_price, pv_dividend)
end

def tickGeneric( ticker_id, tick_type, value)
end

def tickString( ticker_id, tick_type, value)
end

def tickEFP( ticker_id, tick_type, basis_points,
formatted_basis_points, implied_future, hold_days,
future_expiry, dividend_impact, dividends_to_expiry)
end

def orderStatus( order_id, status, filled, remaining,
avg_fill_price, perm_id, parent_id, last_fill_price,
client_id, why_held)
end

def openOrder( order_id, contract, order, order_state)
end

def updateAccountValue( key, value, currency, account_name)
end

def updatePortfolio( contract, position, market_price, market_value,
average_cost, unrealized_pnl, realized_pnl, account_name)
end

def updateAccountTime( time_stamp)
end

def nextValidId( order_id)
end

def contractDetails( req_id, contract_details)
end

def bondContractDetails( req_id, contract_details)
end

def contractDetailsEnd( req_id)
end

def execDetails( order_id, contract, execution)
end

def updateMktDepth( ticker_id, position, operation, side, price, size)
end

def updateMktDepthL2( ticker_id, position, market_maker, operation,
side, price, size)
end

def updateNewsBulletin( msg_id, msgType, message, orig_exchange)
end

def managedAccounts( accounts_list)
end

def receiveFA( fa_data_type, xml)
end

def historicalData( req_id, date, open, high, low,
close, volume, count, pWAP, has_gaps)
# using straight printing
puts 'ruby> id=' + req_id.to_s + ' date = ' + date +
' open=' + open.to_s + ' high=' + high.to_s +
' low=' + low.to_s + ' close=' + close.to_s +
' volume=' + volume.to_s + ' count=' + count.to_s +
' WAP=' + pWAP.to_s + ' hasGaps=' + has_gaps.to_s
end

def scannerParameters( xml)
end

def scannerData( req_id, rank, contract_details, distance,
benchmark, projection, legs_str)
end

def scannerDataEnd( req_id)
end

def realtimeBar( req_id, time, open, high, low, close, volume, wap, count)
end

def currentTime( time)
end

def fundamentalData( req_id, data)
end

# ---------------------------------------------------
# -- methods from AnyWrapper -- must be declared
# ---------------------------------------------------

def connectionClosed()
puts ' [API.connectionClosed] Closed connection with TWS'
end

def error(err_obj_str)
puts ' [API.msg1] ' + err_obj_str if err_obj_str.is_a?(String)
puts ' [API.msg3] ' + err_obj_str.getMessage() if err_obj_str.is_a?(Exception)
end

def error(one, two, str)
puts ' [API.msg2] ' + str + ' {' + one.to_s + ', ' + two.to_s + '}'
end

end # class TestHistData

# *********************************************************************************************
# Running the code
# *********************************************************************************************
x = TestHistData.new()
begin
x.eConnect
puts 'connected ...'
req_id = 0
x.requestHistStockQuote(req_id += 1, 'SPY') # stock
puts 'wait 5 sec'
sleep(5)
x.eDisconnect

rescue Exception => e
puts 'can not connect'
puts 'Exception' + e.message
x.eDisconnect
end


Running it produces the following output :

Server Version:43TWS Time at connection:20090215 23:08:09 ESTconnected ...
[API.msg2] Market data farm connection is OK:usopt {-1, 2104}
wait 5 sec
[API.msg2] Market data farm connection is OK:usfarm {-1, 2104}
[API.msg2] HMDS data farm connection is OK:ushmds2a {-1, 2106}
ruby> id=1 date = 20090211 open=83.44 high=-1.783545891855626e+259 low=84.05 close=4.667261648644354e-62 volume=1079286169 count=-1717986918 WAP=83.53 hasGaps=true
ruby> id=1 date = 20090212 open=82.17 high=1.9991592822007892e+37 low=84.1 close=1.9035983734140052e+185 volume=1079264051 count=858993459 WAP=83.98 hasGaps=true
ruby> id=1 date = 20090213 open=83.55 high=4.667261648670389e-62 low=84.24 close=2.262040992568159e-111 volume=1079285350 count=1717986918 WAP=82.46 hasGaps=true
ruby> id=1 date = finished-20090210 20:00:00-20090213 20:00:00 open=-1.0 high=1.590978778e-314 low=-1.0 close=1.590978778e-314 volume=-1074790400 count=0 WAP=-1.0 hasGaps=false


Compare this with Java version's output (same as displayed earlier on the post, I'm just repeating it here to make it more convenient to compare).

Java> id=2 date = 20090211 open=83.44 high=84.05 low=82.4 close=83.53 volume=2950030 count=855129 WAP=83.4 hasGaps=false
Java> id=2 date = 20090212 open=82.17 high=84.1 low=81.05 close=83.98 volume=4193652 count=1201155 WAP=82.25 hasGaps=false
Java> id=2 date = 20090213 open=83.55 high=84.24 low=82.35 close=82.46 volume=2681140 count=795253 WAP=83.44 hasGaps=false
Java> id=2 date = finished-20090210 20:00:00-20090213 20:00:00 open=-1.0 high=-1.0 low=-1.0 close=-1.0 volume=-1 count=-1 WAP=-1.0 hasGaps=false


Note that id, date, and open values are the same for both Java and JRuby versions. From then on, beginning with high value, the numbers in the JRuby version just look wrong. What's interesting is that JRuby's low value matches up to Java's high value. Maybe there's something wrong with the way JRuby handles all these Java parameters with double and int datatypes?




02/17/09 Update


After some research, I suspect that the incorrect double values passed into historicalData() are a JRuby bug. It appears substantially similar to a reported bug that was fixed in version 1.1.5.

I have opened an issue on the JRuby site http://jira.codehaus.org/browse/JRUBY-3409 and will wait to see what they say...




02/22/09 08:00 Update


There has been no progress on my issue on the JRuby site. To make it easier for the developers there to see the problem, I have reconstructed the error using very simple classes and uploaded the files to their site. Hopefully this will give them what they need to come up with a fix...




02/22/09 22:40 Update


Problem has been solved! The issue has been on my end and I feel bad about having wasted all this time, but hey, lesson learned. The problem was that I had been using NetBeans IDE to perform my development and the JRuby which comes bundled with it is an old version (v 1.1.4). Although I had downloaded and installed JRuby v. 1.1.6, this was NOT the version being run from NetBeans. It was definitely a surprising moment when I discovered the same JRuby code fails when running in NetBeans (JRuby v. 1.1.4), but succeeds when I ran it at the command prompt (JRuby 1.1.6). I have closed the issue on the JRuby site explaining the situation.

To configure NetBeans to use JRuby 1.1.6, go to Tools, Ruby Platforms menu. It will open a new window showing current JRuby version. Here it is showing that it's version 1.1.4:


Click on Add Platform and select 1.1.6 version of jruby.bat that was previously installed. This version should now appear on the Ruby Platform Manager:


Last step is to change the project property by right clicking on the project in Projects window and select Properties. Click on Run in Categories, and change Ruby Platform to 1.1.6.


Finally, here is the JRuby output with correct historical quote:

Server Version:43

TWS Time at connection:20090222 19:41:12 EST

connected ...
wait 5 sec
[API.msg2] Market data farm connection is OK:usopt {-1, 2104}
[API.msg2] Market data farm connection is OK:usfarm {-1, 2104}
[API.msg2] HMDS data farm connection is OK:ushmds2a {-1, 2106}
ruby> id=1 date = 20090211 open=83.44 high=84.05 low=82.4 close=83.53 volume=2950030 count=855129 WAP=83.4 hasGaps=false
ruby> id=1 date = 20090212 open=82.17 high=84.1 low=81.05 close=83.98 volume=4193652 count=1201155 WAP=82.25 hasGaps=false
ruby> id=1 date = 20090213 open=83.55 high=84.24 low=82.35 close=82.46 volume=2681140 count=795253 WAP=83.44 hasGaps=false
ruby> id=1 date = finished-20090210 20:00:00-20090213 20:00:00 open=-1.0 high=-1.0 low=-1.0 close=-1.0 volume=-1 count=-1 WAP=-1.0 hasGaps=false



03/09/09 Update

Upgrading to API v. 9.60
requires addition of these new callback methods.

 
# ---------------------------------------------------
# new callback methods added in API 9.60
# ---------------------------------------------------
def openOrderEnd() # invoked when all orders are sent to a client as a response to reqOpenOrders()
end

def accountDownloadEnd(account_name) #invoked when a complete snapshot of an account state is sent to a client as a response to reqAccountUpdates()
end

def execDetailsEnd( req_id) # invoked when all executions are sent to a client as a response to reqExecutions()
end

def deltaNeutralValidation(req_id, under_comp) # for Delta-Neutral DN RFQ
end

No comments: