Friday, January 30, 2009

Proof Of Concept - 8 of 8 : Stumbling Block

Here's where things get interesting. Now that I'm able to successfully connect to TWS, the next step is to do something useful such as getting current market quote. I found a nice site called StockBotProgramming.com which has sample code to do this in Java. I find that this code is more direct and easier to follow than the one in JavaAPIGettingStarted.pdf.

Some notes about this Java code:
  • Main class implements EWrapper interface
  • A class must implement all of the methods described in the interface
  • The methods in the example is specific to that TWS API, and will not be the same as my version of API (9.51)
  • Methods TickPrice() and TickSize() are callback methods that automatically get called when the quote become available

My task would be to create a class in Ruby which implements this EWrapper interface. More specifically, my version of the EWrapper. So the code will be a little different. Per Java interface definition, all the functions of EWrapper class needs to be declared. The easiest way to do this is to go directly the the EWrapper Java class file, copy, paste into Ruby class and convert to Ruby syntax.

This isn't quite as easy as it looks and I'm getting some strange behavior in the coding. Before handling this issue, I noticed there's a problem with prior post concerning Skipping Incoming connection. So I'm not defining a trusted IP anymore.

In order for readers to follow along, I have to post my code here. This is going to be hard to see. But I've found Google's syntax highlighter tool, so I'll be incorporating that into this blog from now on. The instructions are not too clear. I found this blog article to be much better.

A) This is my test code to request market data:


require 'java'
require 'TwsApi.jar'

include_class 'ib.client.EClientSocket'
include_class 'ib.client.Contract'
include_class 'ib.client.AnyWrapper'
include_class 'ib.client.EWrapper'

# classes for interface method parameters. Not sure if necessary. Can't hurt.
include_class 'ib.client.ContractDetails'
include_class 'ib.client.Execution'
include_class 'ib.client.OrderState'

class TestMktData include EWrapper #note class TestMktData < EWrapper syntax is no longer supported in JRuby 1.0

@mySocket

def initialize
puts 'init'
@mySocket = EClientSocket.new(self)
end

def eConnect
@mySocket.eConnect("127.0.0.1", 7496, 0)
end

def requestQuote
puts 'running requestQuote'
# --------------------------------
myContract = Contract.new
myContract.m_symbol = 'SPY'
myContract.m_secType = 'STK'
myContract.m_exchange = 'SMART'
myContract.m_currency = 'USD'
# options specific below ---
myContract.m_expiry = ''
myContract.m_right = ''
request_id = 1
ticklist = ''
puts 'created new contract, calling reqMktData'
@mySocket.reqMktData(request_id, myContract, ticklist, true)
puts 'sleeping for 10 sec...'
sleep(10)
end

def eDisconnect
@mySocket.eDisconnect
end

#///////////////////////////////////////////////////////////////////////
#// Interface methods
#///////////////////////////////////////////////////////////////////////

def tickPrice( ticker_id, field, price, can_auto_execute)
puts 'in tickPrice'
#puts 'tickPrice() id=' + ticker_id.to_s + ', price = ' + price.to_s
end

def tickSize( tickerId, field, size)
puts 'in tickSize'
#puts 'tickSize() Id: ' + tickerId.to_s + ' Type: ' + field.to_s + ' Size: ' + size.to_s
end

def tickOptionComputation( ticker_id, field, implied_vol, delta, model_price, pv_dividend)
end

def tickGeneric( ticker_id, tick_type, value)
end

def tickString( ticker_id, tick_type, value)
end

def tickEFP( ticker_id, tick_type, basis_points, formatted_basis_points, implied_future, hold_days, future_expiry, dividend_impact, dividends_to_expiry)
end

def orderStatus( order_id, status, filled, remaining, avg_fill_price, perm_id, parent_id, last_fill_price, client_id, why_held)
end

def openOrder( order_id, contract, order, order_state)
end

def updateAccountValue( key, value, currency, account_name)
end

def updatePortfolio( contract, position, market_price, market_value, average_cost, unrealized_pnl, realized_pnl, account_name)
end

def updateAccountTime( time_stamp)
end

def nextValidId( order_id)
end

def contractDetails( req_id, contract_details)
end

def bondContractDetails( req_id, contract_details)
end

def contractDetailsEnd( req_id)
end

def execDetails( order_id, contract, execution)
end

def updateMktDepth( ticker_id, position, operation, side, price, size)
end

def updateMktDepthL2( ticker_id, position, market_maker, operation, side, price, size)
end

def updateNewsBulletin( msg_id, msgType, message, orig_exchange)
end

def managedAccounts( accounts_list)
end

def receiveFA( fa_data_type, xml)
end

def historicalData( req_id, date, open, high, low, close, volume, count, eWAP, has_gaps)
end

def scannerParameters( xml)
end

def scannerData( req_id, rank, contract_details, distance, benchmark, projection, legs_str)
end

def scannerDataEnd( req_id)
end

def realtimeBar( req_id, time, open, high, low, close, volume, wap, count)
end

def currentTime( time)
end

def fundamentalData( req_id, data)
end

end # class TestMktData

# *********************************************************************************************
# Running the code
# *********************************************************************************************

x = TestMktData.new()
begin
x.eConnect
puts 'connected ...'
x.requestQuote
puts 'wait 10 sec'
sleep(10)
x.eDisconnect
puts 'disconnected'
rescue Exception => e
puts 'can not connect'
puts 'Exception' + e.message
x.eDisconnect
end


Line 14: specifies that I'm using a Java interface from TWS API called EWrapper
Line 41: submits market quote for ticker SPY
Lines 54-62: methods that automatically gets called when market data is received
Lines 64-140: the rest of method specifications as required by EWrapper interface
Lines 148-161: runs the code by connecting, requesting market quote, and disconnecting

Running the code produces error:

init
Server Version:43

TWS Time at connection:20090201 22:48:24 EST

connected ...
running requestQuote
created new contract, calling reqMktData
sleeping for 10 sec...
Exception in thread "EReader" :1: #< nameerror::message:0x1de0c09 > (NoMethodError)
...internal jruby stack elided...
from (unknown).(unknown)(:1)

disconnected


What causes this 'stack elided' error? It seems like it has something to do with requesting market data, waiting (sleep), and receiving market data. So I went about commenting out lines of code to see what the behavior is and here's what I've found:

B) Commenting out sleep():

puts 'created new contract, calling reqMktData'
@mySocket.reqMktData(request_id, myContract, ticklist, true)
puts 'sleeping for 10 sec...'
# sleep(10)



produces no errors, but then again, the program didn't wait around to receive quote either:


init
Server Version:43

TWS Time at connection:20090201 23:09:01 EST

connected ...
running requestQuote
created new contract, calling reqMktData
sleeping for 10 sec...
disconnected


C) Can it be that sleep() is the actual problem and has nothing to do with reqMktData()? I tried just connecting, sleeping, and disconnecting:


x = TestMktData.new()
begin
x.eConnect
puts 'connected ...'
# x.requestQuote
puts 'wait 10 sec'
sleep(10)
x.eDisconnect
puts 'disconnected'
rescue Exception => e
puts 'can not connect'
puts 'Exception' + e.message
x.eDisconnect
end


Produces the same 'stack elided' error:


init
Server Version:43

TWS Time at connection:20090201 23:24:53 EST

connected ...
wait 10 sec
Exception in thread "EReader" :1: #< nameerror::message:0xd510e8> (NoMethodError) ...internal jruby stack elided... from (unknown).(unknown)(:1)

disconnected


D) So going back to my previous post on TWS Note - Skipping 'Incoming Connection'. I modify the connect and disconnect code to add the sleep():


require 'java'
require 'TwsApi.jar'
include_class 'ib.client.EClientSocket'

mySocket = EClientSocket.new(self)
begin
mySocket.eConnect("127.0.0.1", 7496, 0) # api
puts 'connected ...'
sleep(10) # this produces error
mySocket.eDisconnect
puts 'disconnected'

rescue Exception => e
puts 'can not connect'
puts e.message
end


Produces 'EReader' error, which is different from 'stack elided' error from before (although consistent with error in post on TWS Note - Skipping 'Incoming Connection'):

Server Version:43

TWS Time at connection:20090201 23:31:00 EST

connected ...
wait 10 sec
Exception in thread "EReader" java.lang.ClassCastException: InterfaceImpl326315435 at ib.client.EReader.eWrapper(EReader.java:47) at ib.client.EReader.run(EReader.java:66)

was connected, now disconnected


So what gives? Why do A and C give 'stack elided' errors and D give an 'EReader' error? Both have sleep() but the difference is that A and C define a new class which implements EWrapper. D doesn't do that. It could be that there are two different error conditions going on. I'm not sure what the answer is yet, but I will look into it some more.

Stay tuned ..

Monday, January 26, 2009

TWS Note - Skipping 'Incoming Connection'

To avoid 'Accept incoming connection?' dialog box from popping up while using API to connect to TWS, simply go to TWS menu. Choose Configure | API | All API Settings...
Then enter 127.0.0.1 as the trusted IP address.




02/01/2009 Update

Changing the configuration above sounds innocent enough. By skipping this dialog box, it saves a mouse click during development, but it seems there is an unintended side effect. Using this very simple to connect and disconnect.

require 'java'
require 'TwsApi.jar'

include_class 'ib.client.EClientSocket'

mySocket = EClientSocket.new(self)
begin
  mySocket.eConnect("127.0.0.1", 7496, 0)  # api
  puts 'connected ...'
  mySocket.eDisconnect
  puts 'disconnected'

rescue  Exception => e
  puts 'can not connect'
  puts e.message
end


Produces this error:

Server Version:43

TWS Time at connection:20090201 15:14:39 EST

connected ...
disconnected
Exception in thread "EReader" java.lang.ClassCastException: InterfaceImpl1026612999
at ib.client.EReader.eWrapper(EReader.java:47)
at ib.client.EReader.run(EReader.java:66)




What would cause this? This error is very consistent and happens at every run. So I went back and changed the trusted IP address to 127.0.0.2 to get the 'Accept incoming connection?' dialog again. The output is now as expected. No errors. So maybe this trusted IP configuration doesn't work after all :(

Server Version:43

TWS Time at connection:20090201 15:24:50 EST

connected ...
disconnected

Friday, January 9, 2009

Proof Of Concept - 7 of 8 : NetBeans Ruby Plugins

Let's review what we have so far for my development environment. Java SDK is installed and IB's Java API successfully compiles in NetBeans. I was able to compile and run IB's test client program and create a JAR file of IB's API. JRuby is installed and runs on top of Java SDK. So now I'm ready to write Ruby code and call IB's Java API.

Before I can do that though, I need to have a development environment for Ruby. Fortunately, NetBeans also supports Ruby development along with Java. To enable Ruby support, go to Tools | Plugins | Available Plugins tab. Choose Ruby and JRuby plugins and click install. It may take a while to fr NetBeans to download all the libraries.


Test out Ruby plugin by creating a Hello World program. Go to File | New Project | Ruby Application. Name it HelloWorld.



Run the application by pressing F6 and 'Hello World' should display in the output window.


Success!! Ruby development is enabled in NetBeans.




Next step is to see if Java classes are accessible. This is a sample Ruby code that calls Java class TreeSet. Note how seamless it is to call Java classes using JRuby.


require 'java'
include_class 'java.util.TreeSet'

set = TreeSet.new
set.add "foo"
set.add "Bar"
set.add "baz"
set.each do |v|
puts "value: #{v}"
end


Press F6 will produce the following in the output window.




Another way of testing this is (just plain JRuby, not NetBeans) to run it via command line. It produces the same output:






02/22/09 Update


JRuby plugin automatically installs version 1.1.4. This is an old version and NetBeans must be manually configured to use version 1.1.6 (installed in Proof Of Concept - 6 of 8). Please see Proof Of Concept 8 of 8b : Historical Quote for how to do this.




Last step is to see if IB's Java API archived file is accessible. Copy file TwsApi.jar (created in Proof Of Concept - 5 of 8) and put it in HelloWorld\lib folder. Run TWS in paper trading mode as this is necessary for for the API to connect to.

Code the following in main.rb. This is really simple code to connect and disconnect from TWS using IB's API.

 
require 'java'
require 'TwsApi.jar'
include_class 'ib.client.EClientSocket'

mySocket = EClientSocket.new(self)
begin
mySocket.eConnect("127.0.0.1", 7496, 0) # api
puts 'connected ...'
mySocket.eDisconnect
puts 'disconnected'
rescue
Exception => e
puts 'can not connect'
puts e.message
end

Press F6 will produce the following output:

Server Version:43TWS Time at connection:20090113 15:27:27 ESTconnected ...disconnected


Running it via command line produces the same output:



At this point, the development environment is set up and Java and API classes are accessible. It is now time to start using IB's API to see its full power.

Proof Of Concept - 6 of 8 : Install JRuby

Go to JRuby home page and download the zip file. As of this writing, the current version is 1.1.6.

  1. I extracted the zip file into C:\jruby\jruby-1.1.6RC1
  2. Added the above path on to the end of PATH environment variable
  3. Add new environment variable JAVA_HOME with value C:\Program Files\Java\jdk1.5.0_17 (this directory was created with installation of Java SDK in Proof Of Concept 3 of 8)
  4. To test whether it installed correctly, run:

    jruby -v

The above command will return the current version.


Proof Of Concept - 5 of 8 : Create Java Archive File (JAR)

A Java JAR file is used for aggregating many files into one. It is generally used to distribute Java classes. All IB's API classes in twsapi\com\ib\client needs to be packaged into .jar file in order to be used in JRuby.

Run NetBeans and choose new project for Java class library. I have named my project TwsApi. Click Finish and the new project is created.
In Explorer, copy com\ib\client folder (created in Proof Of Concept - 4 of 8) and paste it into TwsApi\src folder.
In NetBeans, right click on the project node and select Properties. In Categories: Sources, press Add Folder button and select TwsApi\src\com.


Click OK and the folder is now added to the project. Press F11 to build the project and TwsApi.jar should be created in TwsApi\dist folder.

Wednesday, January 7, 2009

Proof Of Concept – 4 of 8 : IB's Java Test Client

Next step is to use NetBeans to compile and run IB's Java test client. The process is described in documentation JavaAPIGettingStarted.pdf (Chapter 5) on IB's web site. It's a good idea to read this doc and follow the directions. Running the Java test client basically consists of:
  1. Create a new Java application project in NetBeans
  2. Attach packages in Java\com and Java\TestJavaClient folders (see Proof Of Concept - 2 of 8) to the project
  3. Press F6 to run
  4. Test client magically runs and ready to connect to TWS


Well, this didn't quite work out for me. NetBeans was unable to compile the Java program. Note the red exclamation marks in the upper left corner.



Looking a little deeper at the API code, looks like there's some sort of problem with loading the classes in IB's packages. NetBeans just doesn't like the package names on all the Java class files.


I was stuck at this step for a little while but was able to resolve it (with a little bit of hacking). The issue is that NetBeans is unable to load IB's package names com.ib.client and TestJavaClient defined in Java classes in Java\com and Java\TestJavaClient folders. So some editing of the Java class files are required.


1. In Explorer, copy the two folders in Java\ folder and paste them into a new folder. This is where edited versions will be. I just don't like to lose the original version in case something goes wrong. I named this new folder twsapi\.

2. In NetBeans, for each .java file in ib.client package:
  • At the top of the file, change text package com.ib.client; to package ib.client;
  • Save, and there shouldn't be any errors
3. In Explorer, rename folder hierarchy from twsapi\TestJavaClient to twsapi\x\y\TestJavaClient



now becomes






4. In NetBeans, for each .java file in TestJavaClient package:
  • At the top of the file, change text package TestJavaClient; to package y.TestJavaClient;
  • In the rest of the file, change any occurrence of text import com.ib.client. to import ib.client
  • Save, and there shouldn't be any errors

5. Create a new Java application project.

6. Attach packages in twsapi\com and twsapi\x folders to the project. Note no more red exclamation marks in the upper left corner.



7. Press F6 to run.

8. Test client runs and ready to connect to TWS. Try out its functionalities as specified in documentation JavaAPIGettingStarted.pdf.


Tuesday, January 6, 2009

Proof Of Concept - 3 of 8: Install Java SDK and Netbeans

Java Standard Edition Software Development Kit ( SE SDK) and NetBeans IDE must be installed in order to compile and run Java programs. Go to Sun's Java SE Downloads page and locate JDK 5.0 Update 17 with NetBeans IDE 6.5. As of this writing, Java latest release is JDK 6 Update 11. I was going to use this latest release, but after reading through some postings on IB's TWS API forum, some users experienced performance problems with JDK 6 and IB's API. This was not the case with JDK 5. So I've decided to use this the older.

The installation process installs Java SE SDK, Java SE Runtime Environment, and NetBeans IDE in separate folders. Mine are in c:\Program Files\Java\jdk1.5.0_17, c:\Program Files\Java\jre1.5.0_17, and c:\Program Files\NetBeans 6.5. Make a note of these as they will be needed later.

To check installation, run NetBeans and run sample projects that came with it. Just select the project and Run the project (F6).

Proof of Concept - 2 of 8 : IB's API

To install IB's TWS API, go to IB's Programming API page and download the software. I'm using Windows API version 9.51 dated Apr 25 2008. Note that this will install all the APIs: Excel DDE, Java, ActiveX, and C++.

The install is straight forward. Just run the install executable and it does the rest. I've chosen to install mine in the same folder as the TWS client (c:\jts). When done, make note of folders Java\com\ib\client and Java\TestJavaClient.



Java\com\ib\client folder contains all the API classes which will be needed in my trading robot.

Java\TestJavaClient folder contains IB's test application and is used to make sure that basic functionalities of the API works. I will use this code to make sure my development environment is set up properly, but will not need it for the trading robot.

Monday, January 5, 2009

Proof of Concept - 1 of 8 : IB Paper Trading Account

Interactive Brokers does not automatically give customers paper trading accounts.

Log into Account Management. Click on Trading Access | Paper Trading, and follow instructions there. It may take a day or two for the paper trading account to be set up.

Once the account is ready, log into TWS. Red "SIMULATED TRADING" bars on the top and bottom of the application indicates that this is paper trading account. This account will be used for all testing.

Sunday, January 4, 2009

Proof of Concept - Overview

The first thing to do is a proof of concept to make sure that IB API - Java - JRuby will all work together. So a number of installs and sample programs need to be written.

Here is the task list (details of each step to follow in separate posts):

  1. Obtain paper trading user name and password for Trader Work Station. This account will be used for testing.
  2. Install IB's Application Program Interface Software (v 9.51) - This is IB's proprietary API. The trading robot will use these APIs to interact with Trader Work Station.
  3. Install Java SE SDK (v 5.0 Update 17) and Netbeans - Java SDK will enable Java programs and create Java Archive (.jar) files. Netbeans integrated development environment will be used to work with Java and Ruby code.
  4. Compile and run IB's Java test client.
  5. Create Java Archive file (.jar) with IB API classes.
  6. Install JRuby (v 1.1.6) .
  7. Enable Ruby plugins in NetBeans.
  8. Write Ruby programs to test IB API running in JRuby.

Trading Robot Features (v 0.0 initial)

I will work out full features at a later time (after proof of concept works), but the initial idea of the trading robot's features are:

  1. Each position is tied to a specific trading strategy
  2. Position entries are done manually. Note a stock/option position can have one or more legs
  3. Position adjustments are done by the robot
  4. Position exits are done by the robot
  5. Adjustment rules can be customized with parameters and can be reused for different trading strategies
  6. Be able to do the following using IB API:
  • get current quote for stock, index, or option
  • get historic quote - high/low for the past x days
  • get current options info: bid, ask, delta, theta, iv, etc
  • place single and combination orders: limit, market, stop loss, contingent
  • set up alerts
  • detect execution of orders
  • cancel orders

Major components which can be run from a single workstation (Windows XP) are:

  1. IB Traders Work Station (TWS) software - This is the interface that does the actual trading.
  2. Database - To store strategy rules and trading records
  3. Trading Robot program - This program is implemented in JRuby just runs with no user interface. It accesses TWS and the database, issuing orders to TWS and updates the database.
  4. Trading Monitor web application - This program is implemented in Ruby on Rails and provides a view of what the trading robot is doing along with reports

Technologies

I use Interactive Brokers to do my trading because of their rock bottom commissions and speed of execution. Another plus with IB is that they make their proprietary application programming interface (API) available for free. These APIs are available in Excel DDE, Java, ActiveX, C++. Since I had done some work in Java a long time ago, I was interested in the Java API. Well, after looking at Java programming again, I'm not too enthusiastic about doing anything in Java. It's just too syntax intensive and complicated for a part time project.

Last year, I was introduced to Ruby as a programming language and was able to develop a couple of applications using one of its web application frameworks called Ruby On Rails. These small applications that I developed made me appreciate the beauty, simplicity, and power of Ruby language. So this is my programming language of choice for this project.

Having settled on a language, the next question is: Would it be possible to develop my trading robot in Ruby while calling IB's Java API? Well, anything in programming is possible, but would it be feasible? That is the better question. I've found JRuby, which is essentially Ruby implemented in Java. This will make it easy for my Ruby program to access IB's Java API. It will allow me to do my development in Ruby while accessing any Java classes and libraries. Theoretically this should work.

Note that the technologies I've mentioned are all open source. This is important, since the idea behind this project is to be able to develop and deploy it with no cost. Another advantage of using Java and Ruby is that it is supported on many platforms. While I'm working exclusively on MS Windows at this time, I would like to have the flexibility to deploy it on Linux in the future. When I incorporate a database system in the future, it will also be open source.

Introduction

Equity trading is one of my main passions in life. I find that it keeps me extremely motivated, thoughtful, honest, and humble. Since I work full time, it's difficult for me to monitor and adjust my trading positions during market hours. If only I had a computer program to do this job for me, life would be much easier. Trading strategies that I use are systematic (meaning all the steps are well defined), therefore it should be possible to let a program monitor and execute these trades for me. While there are programs and systems that currently do this, they tend to be pricey, geared towards institutions, and do not have the flexibility that I need.

By trade, I'm an Oracle developer. I have extensive experience in designing and developing systems. It is natural that I would want to write my own personal trading robot. This blog is a technical diary of my development process. I will not document my personal trading plan, but I will document the development of this trading robot. This trading system will be implemented in various technologies that I'm not an expert in, such as Java, JRuby, and Ruby. After some research, I found that there is little documentation relating to the integration of these technologies and how they all fit together. It is my goal to document this and related topics as I gain more knowledge. If readers find some of it helpful in developing their own system (which may or may not be related to trading), then I would be happy to have made a positive contribution to the world.