| Overview | Package | Class | Use | Tree | Deprecated | Index | Help |
|
|
|||||||||
| PREV NEXT | FRAMES NO FRAMES | |||||||||
See:
Description
| Packages | |
|---|---|
| com.caplin.config.util | Provides utility classes to aid configuration of Caplin products. |
| com.caplin.config.xml | Provides XML configuration with optional type safety and Java class instantiation. |
| com.caplin.datasrc | The main package for the Java DataSource Developers Kit classes. |
| com.caplin.datasrc.cache | Provides a caching mechanism which allows a DataSource to turn a broadcast source into an active one. |
| com.caplin.datasrc.delay | Provides the ability to create one or more delay channels for sending data to peers after a specified delay period. |
| com.caplin.datasrc.fields | Provides field management interfaces and classes that represent all the configured fields that are available. |
| com.caplin.datasrc.interfaces | Defines the main data interfaces used by DataSource for Java SDK and related products. |
| com.caplin.datasrc.latencychain | |
| com.caplin.datasrc.logging.application | Contains XML Configuration classes to allow configuration of a standard Logger instance for DataSource application logging. |
| com.caplin.datasrc.logging.packet | Provides packet-level logging of DataSource communications with peers. |
| com.caplin.datasrc.mapping.object | Provides object name mapping utility classes. |
| com.caplin.datasrc.udp | Provides the DataSource specific implementation of UDP message send and receive functionality. |
| com.caplin.event | Provides event related functionality such as: an event loop and associated classes for use by timed events or NIO events, and a generic event multicaster. |
| com.caplin.net.udp | Provides access to the UDP command interface for receiving and sending UDP messages. |
| com.caplin.util.base64 | Provides Base64 encoding for sending typed objects using DataSource. |
| com.caplin.util.logging | |
| examples.container | ActiveContainerSource is an example that can provide a Container object
to the Liberator. |
| examples.liberator | |
| examples.liberator.data | |
| examples.liberator.data.filters | |
| examples.liberator.data.impl | |
| examples.liberator.data.translate | |
| examples.liberator.dispatch | |
| examples.multipledatasources | MultipleDataSourcesInOneJVM is an application that shows how to run more than one data source in one JVM. |
| examples.news | NewsSource is an active source that services
requests and discards from a peer. |
| examples.simple | SimpleBroadcast is a very basic broadcast source. |
| examples.sink | Sink is a simple DataSource that listens for a peer connection, then
requests and discards, on a timer, a random selection of objects from its connected peer. |
| examples.source | Source is an active source that services
requests and discards from a peer. |
| examples.udp | UDPControlled is a UDP Controlled broadcast source that provides record
data controlled with various rate parameters. |
The Caplin DataSource for Java SDK (Software Development Kit) is a library of classes that enables developers to create applications that can send and receive financial market data in the form of structured records and fields, pages, news headlines and news stories to and from other DataSource-enabled applications, using Caplin's DataSource protocol.
The kit contains the following components:
Follow the links on the table below for information on the following topics:
A DataSource application can perform the following tasks:
The Java DataSource includes many helper classes and mechanisms for performing these tasks efficiently, including:
Thread. Instead of using Thread.isAlive() to test whether the DataSource is still
running, use DataSource.isRunning(). isRunning() now verifies whether any child threads it may have kicked off are also running.To install DataSource SDK:
To use the DataSource SDK for Java classes:
|
import com.caplin.datasrc.DSMessageListener; import com.caplin.datasrc.interfaces.DSRecord; import com.caplin.datasrc.DataSource; import com.caplin.datasrc.fields.FieldManager; |
A number of example DataSource applications created using DataSource SDK for Java are available in the examples subdirectory. Information about these applications is available here.
Caplin DataSource for Java requires a 1.4 or newer JVM as it uses features only found in these versions. It is tested and approved for use with the Sun 1.4.2 JVM on Windows, Solaris and Linux but should work on any operating system with a suitable JVM installed.
In order to take advantage of configuring the application from an XML stream some changes to the code need to be made.
The old DataSource() constructor that retrieved configuration via System Property has been deprecated.
To upgrade, use the DataSource.DataSource(InputStream) constructor
to take advantage of configuring from any InputStream source. Also remove all System Property
configuration from the code, this is no longer needed. Note that the new constructor
throws IOException and SAXException to indicate issues with retrieving configuration.
Your application receives incoming data from a data provider (e.g. proprietary data feed, database trigger).
This data contains values that need to be contributed to another DataSource application. To be able
to do this, Java DataSource SDK runtime libraries need to be included as a component
within your application. In order to contribute these values, you need to use
the Java classes and interfaces (such as DSData
, DSRecord and DSFactory)
provided by the DataSource library to create data objects and add the values (fields) to these objects.
The object can then be forwarded to where it is needed, be that a Caplin Liberator for onward distribution
over the Internet, or another DataSource application for aggregation or further processing of data.
The application will act as a source of data for any other applications to which it is connected which also use the DataSource protocol. These other applications are known as DataSource peers.
DSData interface
to represent the base data object that is used to create and contribute data.DSRecord, DSPage
, DSNews and DSStory
are extensions which provide specialised functionality for contribution of records, pages and news headlines / stories
respectively. (There are also other extensions of DSPacket
that provide additional packet types such as No Data and Status messages.)Broadcast: A broadcast DataSource is the simplest type of DataSource. It sends all object and updates to all connected peers. If a broadcast DataSource is chosen then when an update is received from the data provider a simple process can be followed to send this data on to connected peers:
DSData (or subclass) object (e.g. DSRecord)
using the DSFactoryDSRecord.addRecordData)DSData.send()
method, or to a specific peer using the DSData.sendToPeer(int peer) method.For an example of a simple broadcast DataSource see the SimpleBroadcast class in the provided examples.
Active: An active DataSource accepts requests for objects, keeps track of which
objects have been requested and only sends updates for those objects. Objects may be discarded
as well as requested. Discarding is when a connected peer tells DataSource that it no longer wishes to receive
updates for this object. Request and discard messages are tracked using the DSRequestListener
and DSDiscardListener respectively.
The following steps should be followed to provide data using an active DataSource:
DataSource.setDSRequestListener(com.caplin.datasrc.DSRequestListener)
and DataSource.setDSDiscardListener(com.caplin.datasrc.DSDiscardListener).DSRequestListener.receiveRequest(int, java.lang.String[]).DSDiscardListener.receiveDiscard(int, java.lang.String[]) method.For an example of an active DataSource see the Source class in the provided examples.
It is also possible to convert a broadcast DataSource to behave as an active DataSource by using the
CacheManager. This provides caching of broadcast data by utilising the
DSSendManager interface to intercept data sent using DSData.send().
See the section on caching for more information on this feature.
It has the advantage of automatically only sending data to peers that have requested it by maintaining a subscription database.
Your application needs to receive data from another DataSource peer to allow monitoring of certain market data elements. This data will be requested from another active peer and if certain conditions are met then some action should be taken.
The application will act as a sink (i.e. consume data, not publish). It will receive data from other peers for processing.
The following steps should be followed to allow data to be received from peers:
DSUpdateListener to the DataSource object using DataSource.setDSUpdateListener(com.caplin.datasrc.DSUpdateListener).DSNoDataListener to the DataSource object.DSMessageListener and
DSStatusListener can be added as required.DSUpdateListener.receiveUpdate(int, com.caplin.datasrc.interfaces.DSData) method.
The first parameter will indicate the peer number that sent the packet, the second parameter will be the DSData packet that has been received.For an example of a DataSource that receives data from another peer see the Sink class in the provided examples.
DataSource applications are configured using XML which can be passed to the DataSource constructor as an InputStream or File. The specification for this is found here: 'DataSource.xml'
The FieldManager can optionally be configured via XML which can also be passed to the DataSource constructor as an InputStream or File. The specification for this can be found here: 'Fields.xml'
There are three main steps required to initialise a DataSource application:
DataSource:
DSMessageListener - Receives callback events when a peer connection
changes state (goes up / down) and also receives system messages from other peers.DSRequestListener - Receives callback events when a connected peer requests
an object from this DataSource. Usually used to begin providing data on demand (active operation).DSDiscardListener - Receives callback events when a connected peer discards
an object from this DataSource and no longer wants to receive updates.DSUpdateListener - Receives callback events when a connected peer sends
a data update for an object that this DataSource is currently receiving.DSStatusListener - Receives callback events when a connected peer sends
a status message for an object. (e.g. indicating an object is stale / not stale).DSNoDataListener - Receives callback events when a connected peer sends
a "no data" message for an object. This usually occurs in response to a request for data from that peer.DataSource.start() method.
This will start the main DataSource thread which will begin to listen for connections and connect to other peers as configured.
import java.util.logging.Logger;
import com.caplin.datasrc.DataSource;
import com.caplin.datasrc.DSMessageListener;
import com.caplin.datasrc.DSUpdateListener;
import com.caplin.datasrc.DSStatusListener;
import com.caplin.datasrc.DSNoDataListener;
public class ExampleDataSource implements DSMessageListener, DSUpdateListener,
DSStatusListener, DSNoDataListener
{
// The main DataSource object
private DataSource dataSource;
// A local FieldManager reference
private FieldManager fieldManager;
// Constructor
public ExampleDataSource(DataSource dataSource)
{
this.dataSource = dataSource;
}
public void receiveMessage(int num, int type, int val, String message)
{
// Add code for receiving messages here ...
}
public void receiveUpdate(int num, DSData dsData)
{
// Add code for receiving updates here ...
}
public void receiveNoData(int num, String subject, int flags)
{
// Add code for receiving nodata messages here ...
}
public void receiveStatus(int num, String name, int flags, int code, String message)
{
// Add code for receiving status messages here ...
}
public static void main(String[]args)
{
// Configuration can be passed in via InputStream or File - here we just pass in a file
DataSource dataSource = new DataSource(new File("conf/DataSource.xml"));
// Initialise the ExampleDataSource object
ExampleDataSource eds = new ExampleDataSource(dataSource);
dataSource.setDSMessageListener(eds);
dataSource.setDSNoDataListener(eds);
dataSource.setDSStatusListener(eds);
dataSource.setDSUpdateListener(eds);
dataSource.start();
}
} |
A DataSource application is bi-directional - a DataSource can send and receive data from another DataSource. In addition, a DataSource can either connect to, or listen for connections from, another DataSource. These other DataSources are referred to as Peers.
The Caplin Liberator and Caplin Transformer are examples of DataSource peers.
Objects can be requested from individual DataSource peers or groups of peers.
An active DataSource is one that will service requests for objects (the opposite of this is a broadcast DataSource, which simply sends all objects and updates to its peers regardless). It is an active DataSource's responsibility to keep track of which objects have been requested and send updates for those objects only.
When a user requests an object, and Caplin Liberator does not already have it, it will request it from one or more of its active sources. If another user requests that object Caplin Liberator will already have all the information it needs, and will respond to the user immediately.
Objects may be discarded as well as requested. This tells the source that we no longer wish to receive updates for this object. When a user logs out or discards an object, Caplin Liberator will send a discard message to the active DataSource (as long as no other user is viewing that object). This discard will actually take place one minute after the user discarded the object; this prevents objects being requested and discarded from the source unnecessarily.
The DSData interface is the base interface used for sending
data updates to DataSource peers. Data is passed using the format of an object name
(subject) to identify the object and one or more field number/value pairs. Typically a
field number will map to field name such as "Bid" and "Ask".
DSData provides numerous methods such as
DSData.addData and
DSData.addBinaryData to add field pairs to objects,
and other methods to access and change data values.
The DSFactory
Example code:
| DSRecord record = DSFactory.createDSRecord(dataSource, "/I/VOD.L"); record.addRecordData("Bid", 12.385); record.addRecordData("Ask", 12.398); // Send data to all connected peers. record.send(); |
DSRecord, DSPage,
DSNews and DSStory
extend DSData
and offer more methods for adding data that are specific to certain object types.
DSRecord uses field mappings
(see Field XML Schema) to map field names to numbers and vice versa.DSPage allows text to be added using a column/row index.DSNews represents a news headline and provides helper methods for
headline and story links, and the ability to add different types of news codes.DSStory represents a news story and provides a helper method for
setting the story text. A news story can be created with either a CREATE or APPEND action.DSField represents a field/value pair and can
be accessed through DSData and sub-classes to get or
set its value. The DSField also provides access, if present, to its associated
FieldInfo instance as provided by the FieldManagerIterating over fields in a DSData, Example code:
Iterator i = data.iterator();
while(i.hasNext())
{
DSField field = (DSField)i.next();
System.out.println("FieldNum="+ field.getFieldNumber()+" Value="+field.getValue());
} |
A DataSource can request or discard objects from a peer by using the RequestDiscardOperations class.
Use the DataSource.getRequestDiscardOperations()
method to get an instance of this class for a DataSource. A DataSource peer can listen
for requests and discards by adding implementations of the DSRequestListener and
DSDiscardListener respectively.
Example code
DataSource dataSource = new DataSource(new File("conf/DataSource.xml"));
String[] recordSubjects = { "/INTL.O", "/SUNW.O", "/MSFT.O", "/CSCO.O", "/DELL.O" };
//Get an instance of RequestDiscardOperations
RequestDiscardOperations ops = dataSource.getRequestDiscardOperations();
//Make a request to all connected peers
ops.requestObjects(recordSubjects);
// ... some time later...
// Make a discard to all connected peers
ops.discardObjects(recordSubjects);
|
A DataSource can listen for data updates from connected peers by registering a
DSUpdateListener interface using the DataSource.setDSUpdateListener(com.caplin.datasrc.DSUpdateListener) method.
The DSUpdateListener.receiveUpdate(int, com.caplin.datasrc.interfaces.DSData) method will be invoked on the listener each time data is received from a peer.
A DataSource application can also receive contributions made by RTTP clients connected to a Liberator. This behaviour is often desired if these user contributions need to be passed back to an upstream component, for example in the case of a trade message. To achieve this the DataSource must be configured to send the CONTRIB flag, see the local tag in the DataSource XML Configuration Schema for more details.
The Liberator will send any client contributions to objects that are owned by a particular DataSource back to that DataSource. An object is owned by a DataSource if that DataSource has provided data for the object in the current Liberator session. If more than one DataSource provides data then both will receive the client contribution.
The DSStatus interface represents a message
indicating the current status of an object. These can be sent to peers or received from peers.
To create a DSStatus object use the DSFactory.createDSStatus(com.caplin.datasrc.DataSource, java.lang.String, int, int, java.lang.String) factory method.
Example code
| DSStatus status = DSFactory.createDSStatus(dataSource, "/DEMO/TEST",
S_STALE, 15, "Stale due to feed down"); status.send(); |
A DataSource will receive these messages through the DSStatusListener interface
registered on the DataSource object using DataSource.setDSStatusListener(com.caplin.datasrc.DSStatusListener) method.
System status messages may be sent using the StatusOperations class.
The DataSource peers can listen for these messages by adding a DSMessageListener
to the DataSource. The StatusOperations class also provides methods that allow
a DataSource to disconnect and stop listening for messages (StatusOperations.setStatusDown())
and subsequently reconnect and start listening again (StatusOperations.setStatusUp()).
With the StatusOperations.setStatusMessage(int messageId, String message) function it is possible
to send a status message to all peers.
Example code:
StatusOperations statusOperations; statusOperations = dataSource.getStatusOperations(); statusOperations.setStatusMessage(1, "Going down in 1 minute..."); // Code to wait a minute ... statusOperations.setStatusDown(); // Do some other operations until ready to set status up again statusOperations.setStatusUp(); statusOperations.setStatusMessage(2, "Back up again"); |
Systen status messages can be sent to a particular peer. All three setStatus functions have equivalent ones that take additional
parameter: peer number which is an index in the peers array in DataSource object. See: StatusOperations.setStatusDownToPeer(int peer),
StatusOperations.setStatusUpToPeer(int peer),
StatusOperations.setStatusMessageToPeer(int peernum, int messageId, String message).
Example code:
// all operations on peer 0 only StatusOperations statusOperations = dataSource.getStatusOperations(); statusOperations.setStatusMessage( 0, 1, "Going down in 1 minute..."); // Code to wait a minute ... statusOperations.setStatusDown(0); // Do some other operations until ready to set status up again statusOperations.setStatusUp(0); statusOperations.setStatusMessage(0, 2, "Back up again"); |
The Container object DSContainer represents a collection of objects that are related in some way.
When sent to a Liberator, the Container allows a single subscription by a client to subscribe to a container and therefore its contents.
Any updates for the linked objects will be automatically sent to the subscribers of any containers the object belongs to.
New (linked) objects can be easily added to the Container using the
DSContainer.addObject(String) method.
Objects can also be removed from the Container using the
DSContainer.removeObject(String) method.
If a container is sent with the DSData.F_IMAGE flag, the cached container is cleared and replaced with any objects
added in this container update. Otherwise, adding or removing merely adjusts the existing cached container.
Note that 'adding' a DataSource object to a container actually adds a link to that object. So for example a Container object called /BOOK1 could contain /OBJ1 and /OBJ2, where /OBJ1 is a record data type and /OBJ2 is a news data type.
Example code:
// Create a container with two constituents
...
DSContainer container = DSFactory.createDSContainer(info.getDataSource(), info.getName(), getFlags());
container.addObject("/RECORD/MSFT");
container.addObject("/NEWS");
container.send();
|
Example code:
// Change the contents of an existing container
...
DSContainer container = DSFactory.createDSContainer(info.getDataSource(), info.getName(), getFlags());
container.removeObject("/RECORD/MSFT");
container.addObject("/RECORD/ORCL");
container.send();
|
DataSource's packet log file keeps
a record of all data and to which destinations (peers) the data was sent.
The packet log is stored in a proprietary compressed binary format and requires the use of
JLogCat to view it as text. The packet log is configured
using the packetLog tag in the DataSource XML Configuration file.
See here for more details on the XML configuration.
File log wrapping functionality allowing to configure how the packet log files are being cycled, more details in packetLog tag description.
The Datasource text log is for application logging, and writes to an instance of a java.util.logger.Logger class.
DataSource includes a default java.util.logger.Handler that writes to a file and formats all messages with a date-stamp,
log level and performs parameter substitution using java.text.MessageFormat. To access the logger at runtime, either
use the static java.util.logger.Logger.getLogger(String name) method to access using the subsytem attribute specified in the
logging tag or alternatively use Configurables.getLogger() method.
The text log is configured using the logging tag in the DataSource XML Configuration file. See here for more details on the XML configuration.
Rotating Logs
ConfigurableRotateLogHandler is a configurable Handler.
The class will log to a file and after a user specified number of hours, the current log will be archived as
current_log_name.sequence_number
and a new log will be created with the name
current_log_name. All logging messages will then be written to this file. The sequence number starts at 1 and is then incremented
after each archive file is created. By default, the class will rotate after 48 hours.
Although this class is configured by hours, it may take longer than the specified time to rotate the logs as the check to rotate the logs is only performed when a log message is passed to the class for writing out to a file.
This functionality is enabled through the DataSource XML configuration file.
Example code:
<dataSource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" port="2500" id="1" >
<logging default="true" defaultFileName="logs/sample.log" level="FINE"
logRotateHours="24" useParentHandlers="false" />
...
</dataSource>
|
The class can be used by defining an instance of it and then using the Logger.addHandler(Handler handler)
method. The code below shows how this can be done.
//define a logger and initialise it
Logger logger;
...
//create an instance of the class. By default it will rotate every 48 hours
ConfigurableRotateLogHandler handler = new ConfigurableRotateLogHandler(filename);
logger.addHandler(handler);
logger.log(Level.INFO,"Logger created successfully");
|
DataSource uses the standard Java security packages to provide message transportation under SSL.
The standard java.net.ssl.<property> is used to point to truststores/keystores and to fine-tune settings. See http://java.sun.com/j2se/1.5.0/docs/guide/security/jsse/JSSERefGuide.html for further details.
It is possible to run more than one datasource in one JVM. Possible use cases for this are for the client application to manage the datasource by controlling startup and shutdown and testing that two datasources are communicating correctly
Simply costruct two datasources with the required configuration
...
DataSource dataSourceOne = new DataSource(new File(datasourceOneConfiguration), new File(fieldsConfigOneLocation));
DataSource dataSourceTwo = new DataSource(new File(datasourceTwoconfiguration), new File(fieldsConfigTwoLocation));
...
|
An example application that runs two datasources is provided in the examples.multipledatasources package.
When running with multiple datasources there are a few things to note:
rmiClientPort will be the only one used as further ones are not required. Please note however, that each
DataSource does use an independent JNDI server and thus requires and uses different values for their JMXManager's rmiRegistryPort.
DataSource provides a number of helper classes and features for commonly used functionality.
DSPacket.send() method can be intercepted by a custom handler and any desired behaviour can be added.
This mechanism forms the basis for the caching and delaying
features described below.
To add a custom send handler, an implementation of the DSSendManager interface
should be created. This DSSendManager is put in place by calling DataSource.setDSSendManager(com.caplin.datasrc.DSSendManager).
Once in place, all DSData.send() calls are intercepted by the DSSendManager's DSSendManager.send(com.caplin.datasrc.interfaces.DSPacket)
method. Any custom behaviour can then be added, for example delaying the sending of data or storing the packet, etc.
Example:
public class SendAndStore extends SendDirect
{
public SendAndStore(DataSource dataSource)
{
super(dataSource);
}
public void send(DSPacket packet)
{
super.send(packet);
// store the packet contents somewhere for later use
}
}
|
Setting the DSSendManager:
// ... during initialisation code - have an instance of DataSource called dataSource DSSendManager sendAndStore = new SendAndStore(dataSource); dataSource.setDSSendManager(sendAndStore); |
The com.caplin.datasrc.cache package provides a pre-built cache
that enables a broadcast DataSource (or a DataSource receiving upstream data in a broadcast manner)
to act as an active source to its peers without the need to implement caching and subscription
management.
The CacheManager registers itself
with the DataSource as the DSSendManager, DSRequestListener
and DSDiscardListener. It will then cache any DSData sent and provide
subscription management, only sending data to peers that have it currently requested.
The CacheManager can be configured using the <activeCache> tag in the DataSource XML configuration file. See here for more details on the XML configuration.
The FieldManager provides field name, number and index
mapping and configuration. It allows interchangeable use (and conversion of) string field names,
integer DataSource field numbers and a direct index into the underlying data structure.
Each field is represented as a FieldInfo instance, which
provides access to the name, field number and index for the field.
The FieldManager is particularly useful when dealing with DSRecord
objects.
The FieldManager is configured at DataSource initialisation time using an XML configuration file (see here for details.
Fields can be added dynamically at runtime using the
FieldManager.addField(java.lang.String, int) methods
The com.caplin.datasrc.delay package provides data-delay functionality. This allows for
easy configuration of any number of delay channels that will provide data with a configurable delay
period. Each delay channel will delay, by a configurable period, all data that is sent using
DSPacket.send() and also prefix all object names with a configurable prefix.
For example, a prefix of "/D/" would mean that every delayed object would be prefixed "/D/",
differentiating it from the real-time feed and any other delay channels. Delayed data can also be sent in
configurable batches to make it more efficent.
The DelayManager intercepts all sent data and can be configured
to either block or send the live data and then add any number of delay channels with data prefixes (as detailed above).
The DelayManager and delay channels are configured using the delayManager and
delayChannel tags in the DataSource XML
Configuration file. See here for more details on the XML configuration.
DataSource provides an object name mapping feature that allows any number of pattern-based mappings to be configured.
For example, the pattern *.FX -> /FX/* will map any incoming data that matches *.FX to /FX/* (e.g. the incoming data object name USDGBP.FX will be sent out to peers as /FX/USDGBP).
Object name mapping using pattern matching is enabled by using the patternManager tag in the DataSource XML Configuration file. Individual patterns are added with instances of pattern tags. See here for more details on the XML configuration.
DataSource provides a generic XML configuration mechanism that allows Java Objects to be created and configured
from schema-defined XML files. This mechanism is used to configure the DataSource object itself,
all of its utility classes such as UDPSignal and
also theFieldManager class via the Fields XML.
The class used to create and configure Java objects is XMLConfigurationContext. It
uses SAX-based callback parsing of the XML, and can be extended to provide custom functionality. It
will create a named class using a default constructor, then use Java-Bean standard setters to configure it (e.g.
set[AttributeName] or set[ElementName] or add[ElementName]. The available standard Schema types are defined in
XMLSimpleTypes and their associated Java Object types or primitive types are available
from the XMLSimpleTypeMapper.
The example applications use this type of XML configuration provide a good starting point for using XML configuration in your own application
This functionality allows the DataSource to read a log file containing previously received data from other peers and process it in a similar way. To configure and use the replay manager add the <replayManager> tag to the DataSource XML configuration file. The available replay manager modes (depending on the replayType parameter value) are:
| replayType value | Optional parameters | Description | ||||
| "all" |
|
To replay all updates from the latest log file on start-up. | ||||
| "date" |
|
To replay updates received after specified date only. There are two optional parameters: dateString to specify the date to replay updates from and dateFormatClass the name of the class to format the given date. The provided class should be a subclass of a the java.text.DateFormat class with a method format(...) overridden. Otherwise or if there is no dateFormatClass parameter a standard ´dd/MM/yy HH:mm´ pattern and the standard java.util.SimpleDateFormat class are used. | ||||
| "fieldValue" |
|
To replay all updates received after the first occurrence of the specified field/value pair. | ||||
| "selective" |
|
To replay only selected updates filtered on: a symbol name, a field and its value. Omitting one or more parameters will result in increasing the range of the replayed data - for example <replayManager replayType="selective" symbol="/DELL"/> would replay all updates for symbol "/DELL". |
<replayManager replayType="all" /> |
<replayManager replayType="date" dateString="2004-Dec-08 21:01" dateFormatClass="com.company.DateFormatImpl"/> |
<replayManager replayType="selective" symbol="/DELL" fieldNum="25" fieldVal="ABC"/> |
<replayManager replayType="fieldValue" fieldNum="25" fieldVal="133"/> |
<replayManager replayType="all" fileName="packet-7.log"/> |
All the replayed updates will be received by the default update listener.
To replay a log file while the DataSource is running
(rather than to start-up replay from XML configuration) and/or provide
an update listener different than the default one there is a group
of methods in the ReplayManager class:
replayManager.replay( DSUpdateListener updateListener, String fileName ); |
replayManager.replayByDate( DSUpdateListener updateListener, String fileName, Date date ); |
replayManager.replayByFieldValue( DSUpdateListener updateListener, String fileName, int fieldNum, String value ); |
replayManager.replaySelected( DSUpdateListener updateListener, String fileName, String symbol, int fieldNum, String value ); |
A DataSource can be configured to send and receive UDP messages. This allows for basic management of a DataSource by an administrator, or control of another application by the DataSource. A typical function performed with a UDP message would be resetting the DataSource's peer connection to its primary peer after a failover event.
The UDPSignal class provides this functionality. It is enabled and configured to
listen on a particular interface/port with the udpSignal tag in the DataSource XML
Configuration file.
For each UDP Command that the DataSource wishes to process, a UDPCommandListener
should be added to the UDPSignal using the MulticastingUDPAccessor.addUDPCommandListener(java.lang.String, com.caplin.net.udp.UDPCommandListener) method. The
UDPCommandListener will be notified if a UDP message is received
that has a first word that matches the String that it was registered with.
To send a UDP message (either to the DataSource itself or to another host / port), one of the
MulticastingUDPAccessor.send(java.lang.String) methods should be used. Note: When using this
method the message will be sent 3 times due to the inherent unreliability of the UDP protocol. The DataSource will filter
out these duplicates when it receives the UDP message, but other applications may not.
This DataSource for Java SDK comes with a number of example applications. They show how to use the main features of the SDK. They also show how to extend the XML configuration mechanism to provide type safe construction and configuration of Java classes.
Each of the examples can be run from within its own directory (in the default installation) by specifying the examples.jar, datasrc.jar and xercesImpl.jar in the classpath.
E.g. To run the SimpleBroadcast example, make sure you are in the examples/simple directory and issue the following command (note - the ; separator in the classpath may vary depending on operating system.):
java -classpath ../examples.jar;../../lib/datasrc.jar;../../lib/xercesImpl.jar examples.simple.SimpleBroadcast |
Click on the appropriate package link for more details on the chosen example.
examples.simple - A simple broadcast DataSource, which supplies a fixed list of objects on connection to a peer.examples.source - An active DataSource that provides data from its XML-configured ObjectDataBase on demand.examples.sink - A consumer of data - will request a list of objects from its connected peer, use in tandem with examples.source.Sourceexamples.news - An example DataSource providing news headlines and stories.examples.udp - A UDP Controlled example providing a configurable list of record data with
various parameters, such as data-rate, controllable at runtime using UDP messages.examples.container - An example that uses the container object to model order books.examples.multipledatasources - An example of using more than one DataSource in a JVM.Datasource for Java supports latency chaining both as a Latency chain source and as a Latency Chain link.
A latency chain is a way of measuring the latency of any given message as it
passes through connected systems. When the message is originated at the first
datasource, the message is timestamped.
As the packet is transmitted from datasource to datasource (and on through the
liberator), each datasource can record an entry and exit timestamp difference.
This allows the end user to see the latency at each point in the system
If your datasource should be the start of the chain, at the point at which it uses the DSFactory method to create a DSData, it should call LatencyChain.initLatency, passing in the new DSData.
In order to maintain good records for a latency chain, you should record both an
Entry and an Exit time.
Luckily, the implementation of DataSource for Java will do this for you. Just before
you receive an Update, DS4J will stamp the packet with an Entry time, and just before
it is sent to any peers, the packet will be stamped with an exit timestamp.
These actions will only happen if the packet already has a Latency Init field present
and if the configuration has enabled latency chaining.
Since version 3.4, a DataSource is configured using an XML file. This replaces the previous configuration mechanism of passing a Properties object in to the DataSource constructor. Note: For backwards compatibility the Properties mechanism is still available, though not recommended. There is no guarantee that it will be supported in future versions of the DataSource SDK.
To use the XML configuration, pass it in via a File or InputStream to the DataSource constructor.
DataSource for Java SDK uses XML files validated against XML schemas. This enforces order, so elements within other elements must appear in the correct order as declared in the schema. The hierarchy and order of tags is shown below.
N.B. Tags without children must be self-closing (end with />) or the parser will throw an error.
<dataSource>
<logging />
<udpSignal />
<packetLog>
<pattern />
</packetLog>
<replayManager />
<latencyChainManager />
<patternManager>
<pattern />
</pattern>
</patternManager>
<activeCache />
<delayManager>
<delayChannel>
</delayChannel>
</delayManager>
<peer>
<heartbeat />
<local />
<remote />
<destination />
</peer>
<jmxManager />
<sendDirect />
</dataSource>
A description of each tag and its attributes follows.
The root tag for configuring a DataSource.
Configures how peers will identify and connect to this DataSource application. This application will identify its peers using the information set in the <peer> tags.
Attributes:
| Name | Type | Default | Required | Description |
| appName | string | [class name] | No | The name of this DataSource, and how DataSource peers will identify it. This name can be overridden by putting a value in the name attribute of a <peer> element. |
| id | int | 0 | No | ID number of this DataSource. This ID can be overridden by putting a value in the id attribute of a <peer> element. It must match the id attribute given in the <remote> element of the remote DataSource's configuration |
| address | string | localhost1 | No | Network interface to listen for connections from DataSource peers |
| port | int | 0 | No | Network port to listen for connections from DataSource peers. The default of 0 means that no connections can be made to this DataSource |
| SSLAddress | string | localhost | No | Network interface to listen for SSL connections from DataSource peers |
| SSLPort | int | 0 | No | Network port to listen for SSL connections from DataSource peers. The default of 0 means that no SSL connections can be made to this DataSource |
| peerConnectionTimeout | int | 10 | No | The amount of time to wait for a valid peer info packet (protocol negotiation) when making or receiving a peer connection. |
<dataSource> tags can contain the following elements:
|
<logging> |
(either 0 or 1 occurrence) |
| (either 0 or 1 occurrence) | |
| (either 0 or 1 occurrence) | |
| (either 0 or 1 occurrence) | |
| (either 0 or 1 occurrence) | |
| (either 0 or 1 occurrence) | |
| (either 0 or 1 occurrence) | |
|
(either 0 or 1 occurrence) |
|
|
<peer> |
(either 0 or up to 63 occurrences) |
|
(either 0 or 1 occurrence) |
Example:
| <dataSource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" port="25099" appName="TEST"> <logging default="true" level="ALL" useParentHandlers="false" /> <udpSignal port="2459" /> <peer> <local id="112" type="ACTIVE CONTRIB"/> <destination address="merlin" port="25007" /> </peer> </dataSource> |
Configures how your DataSource application logs internal status messages and error messages.
This tag will result in a Logger instance being created with the specified configuration.
This logger instance is available from Configurables.getLogger().
Attributes:
| Name | Type | Default | Required | Description |
| subsystem | string | "com.caplin.datasrc.DataSource" | No | The name for the Logger (see java.util.logging.Logger) |
| default | boolean | true | No | Specifies whether the default DataSource implementation of java.util.logging.Handler is used publishing log messages. The default handler writes to a text file. Each line writes out the date in ISO8601 format, the level of the message and the message itself. |
| defaultFileName | string | datasource.log | No | If using the default Handler (i.e. default attribute set to true), sets the name of the output file. |
| level | string | "INFO" | No | Sets the java.util.logging.Level for DataSource logging.
This will pick up any custom levels that have been added as well as the
standard levels. The standard Java levels for logging messages are:
The logger will inherit the system default of any logging properties. |
| useParentHandlers | boolean | false | No | Specifies whether or not this logger should send its output to its parent Logger. |
| logRotateHours | integer | 48 | No | Specifies the period of time in hours before the log is wrapped (i.e. the current log is closed and renamed and a new log is created). If this attribute is left out, then the logs will not wrap. |
| count | integer | 10 | No | Specifies how many wrapped logs to keep, including the currently used log. If this attribute is left out then 10 logs files will be maintained provided the logRotateHours attribute is specified. Otherwise there will only ever be one log file. |
Example:
| <logging default="true" level="ALL" useParentHandlers="false" /> |
Configures how your DataSource application sends and receives UDP commands. The kit includes a UDP command interface that enables you to receive and send UDP messages in order to perform management operations such as resetting peer connections after failover.
See the section on UDP Messages for more detail on how to use this functionality.
This tag will result in a UDPSignal instance being created with the specified configuration.
This UDPSignal instance is available from DataSource.getUDPSignal().
Attributes:
| Name | Type | Default | Required | Description |
| address | string | 127.0.0.1 | No | The interface to bind to. |
| port | integer | 0 | Yes | The port to listen to for a UDP signal. |
Example:
| <udpSignal port="2459" /> |
Configures how your DataSource records binary messages sent between this DataSource and its peers.
This tag will result in a packet-level log file being created with the specified configuration, unless the noPacketLog attribute is set.
Attributes:
| Name | Type | Default | Required | Description |
| append | boolean | false | No | If true, append to the end of the existing packet log file. If false, start a new file. |
| cyclePeriod | integer | 1440 | No | The frequency (in minutes) of log files rotation tries. The try fails only if the size of the actual log file is less than minSize parameter. Setting cyclePeriod to 0 will prevent any log rotation tries apart from the ones based on exceeding maxSize parameter by the actual log file size (see maxSize parameter description below). The default value is one day. |
| cycleTime | integer | 240 | No | If cycleTime is < 1440 (one day) it will result in the cycling of the packet log beginning at the time of day specified (in minutes), and occurring every cyclePeriod minutes (subject to the minimum size specified by the minSize parameter being met). The default value is 240 which, with the default cyclePeriod of 1440, means the logs are cycled at 4:00am every day. If cycleTime is > 1440 (one day) it will result in the cycling of the packet log beginning at the time since the start of the week specified (in minutes), and occurring every cyclePeriod minutes (subject to the minimum size specified by the minSize parameter being met). For example: if cycleTime=2880 (two days) and cyclePeriod=10080 (a week) the logs cycle (subject to minSize parameter ) every Wednesday at 0:00. |
| maxSize | integer | 0 | No | The maximum size in bytes of the packet log before rotating, or 0 for no limit. If the size of the log file is greater than this parameter it will force the rotation, regardless of the all other parameters values. This parameter can be used to put absolute maximum limits on log size. |
| minSize | integer | 0 | No | The minimum size in bytes of the packet log that will allow a periodical rotation to occur (see cyclePeriod parameter), or 0 for no minimum. |
| count | integer | 10 | No | The maximum number of old log files (with the same name prefixes) to keep when rotating the logs. |
| pattern | String | No | Deprecated The pattern to name the log file. If the <pattern> tag is used then this attribute is ignored. | |
| noPacketLog | boolean | false | No | If true, suppresses writing out a packet log. |
Example:
| <packetLog append="true" maxSize="100000" minSize="10000" cyclePeriod="3600" count="10"/> |
Pattern tag is optional and there can be only one in a <packetLog> tag. It allows to specify the name of the log files to be created by the DataSource. The pattern may be constructed with an expression for java.text.SimpleDateFormat class to parse, so the file names would contain the actual date and/or time. There are several special character allowed:
| Name | Type | Default | Required | Description |
| patternType | string | "text" | No | Possible values: "text" - patternString parameter contains a string to name the log file with, "date" - patternString parameter contains date pattern for the java.text.SimpleDateFormat class. |
| patternString | string | "packet.log" | No | Contains the pattern to name the log file. With patternType="date" it should be in java.text.SimpleDateFormat acceptable format. |
Example:
<packetLog append="true" maxSize="100000" minSize="10000"cyclePeriod="3600" count="10"> <pattern patternType="date" patternString="'%u/packet-%n-'yyyy-MM-dd'.log'" /> </packetLog> |
Tag for replaying data on DataSource startup.
Attributes:
| Name | Type | Default | Required | Description |
| replayType | string | No | The type of replay, possible values: "all", "date", "fieldValue", "selective". | |
| dateString | string | No | Date/time pattern for the dateFormatClass class. (Applies to replayType="date" only, ignored otherwise.) |
|
| dateFormatClass | string | No | The name of a class to format dateString parameter. Should be a subclass of the
java.text.DateFormat class. If not, or if this parameter is not specified the
standard java.text.SimpleDateFormat is used. (Applies to replayType="date" only, ignored otherwise.) |
|
| symbol | string | No | The name of the symbol to replay. (Applies to replayType="selective" only, ignored otherwise.) |
|
| fieldNum | integer | No | The number of the field to replay. (Applies to replayType="fieldValue" or replayType="selective"only, ignored otherwise.) |
|
| fieldVal | string | No | The value of the field. (Applies to replayType="fieldValue" or replayType="selective"only, ignored otherwise.)/td> | |
| fileName | string | No | The name of the packet log file to replay (e.g. "logs/packet.log"). If this attribute is not specified then the most recent packet log file that was written is replayed. (Applies to every replayType.) |
Example:
| <replayManager replayType="date" dateString="2004-Dec-08 21:01"
dateFormatClass="com.company.DateFormatImpl" fileName="packet.log"/> |
Tag for enabling latency chaining.
Attributes:
| Name | Type | Default | Required | Description |
| InitialTimeStampField | string | 1 | No | Sets the rttp field number for the InitialTimeStamp |
| TimeStampListField | string | 2 | No | Sets the rttp field number for the TimeStampList |
| EventListField | string | 3 | No | Sets the rttp field number for the EventList |
| Base64 | string | ignore | No | Sets the configuration for Base64 encoding of the fields.
|
Example:
| <latencyChainManager InitialTimeStampField="2" TimeStampListField="3"
EventListField="4"/> |
Parent tag for replaying data on DataSource startup. A DataSource application can be configured to map its internal namespace to a different external namespace. This can either be used to simply make it a valid object name for an external source, or to create a complex directory structure of objects.
A valid pattern contains either 0 or 1 * characters, where a * represents 0 or more characters to match, much like a filename pattern match.
If present, patterns will be applied in the following ways to different actions:
Forward Mapping (e.g. if object name matches "from", will be mapped to the "to" form.)RequestDiscardOperations class.DSData (or subclasses) packet subject names for packets to be sent to peers.DSUpdateListener.DSRequestListener
and DSDiscardListener classes.DSData (or subclasses) packet subject names.This tag takes no attributes.
This tag will result in a PatternManager instance being
created with Pattern objects for each
<pattern> child tag.
The PatternManager instance is available from DataSource.getPatternManager().
<patternManager> tags can contain the following elements:
|
<pattern> |
(unlimited occurrences) |
Example:
| <patternManager> <pattern from="*.FX" to="/FX/*" /> <pattern from="MSFT*" to="/Microsoft/*" /> <\patternManager> |
Configures how DataSource maps object names that match the pattern specified.
A Pattern object is created and added to the PatternManager
for each instance of this tag.
Attributes:
| Name | Type | Default | Required | Description |
| from | string | [no default] | Yes | The name to search for |
| to | string | [no default] | Yes | The name to change to. |
Example:
| <pattern from="*.FX" to="/FX/*" /> |
Configures the application's Active Cache. Active Caching enables a broadcast-style
data feed to be turned into an active request-based one, by caching all data
in DataSource and handling requests for that data internally (see here for a
description of this feature). The Active Cache makes
it unnecessary to write a database or caching mechanism yourself, which is
useful when one DataSource is feeding multiple DataSource peers, each of which is
only interested in a small subset of the overall data.
This tag will result in a CacheManager instance being created with the specified configuration.
The CacheManager instance will automatically add itself to the DataSource and act as the Active Cache.
Note: Use of both the Active Cache and DelayManager simultaneously is not supported, nor is a custom DSSendManager supported if using one of these features.
| Name | Type | Default | Required | Description |
| enable | boolean | true | No | If true, turns on the active cache |
| hashSize | integer | 1024 | No | Number of entries in the internal hashtable used for storing the active objects. |
| symbol | XML tag | [no default] | No | The symbol to pre-request from the application on DataSource startup |
Example:
<activeCache enable="true" hashSize="2048">
<symbol symbolName="/DELL"/>
<symbol symbolName="/MSFT"/>
</activeCache> |
Configures a symbol (object name) to be pre-requested on DataSource start up from the application by the CacheManager and added to the cache.
Attributes:
| Name | Type | Default | Required | Description |
| symbolName | string | [no default] | Yes | The name of the object to pre-request. |
Example:
| <symbol symbolName="/DELL"/> |
Configures the DataSource application to send (or resend) its data after a specified time as a "Delay Channel". Any number of delay channels can be created with a defined prefix for the object name and a defined delay time. The live channel can remain or can be blocked so that all data is delayed. In addition, the delayed updates can be batched for efficiency.
This tag will result in a DelayManager instance being created
with DelayChannel objects for each
<delayChannel> child tag.
Note: Use of both the Active Cache and DelayManager simultaneously is not supported, nor is a custom DSSendManager supported if using one of these features.
Attributes:
| Name | Type | Default | Required | Description |
| blockRealTime | boolean | false | No | Determines whether to block the "live" undelayed channel. If false, both the real-time and delayed information is sent. |
| batchTime | long | 500 | No | Default time in milliseconds to send batches of delayed data. Each <delayChannel> element can override this value. |
| batchSize | int | 10 | No | Number of updates to be included in each batch for delay channels. It is recommended that this value is set to the number of updates expected in the batchTime attribute. This may be overridden on a channel-by-channel basis. |
<delayManager> tags can contain the following element:
|
(unlimited occurrences) |
Example:
| <delayManager> <delayChannel prefix="" delayTime="1000" batchTime="500" batchSize="50" /> <delayChannel prefix="/TEST1" delayTime="5000" batchTime="1000" batchSize="100" /> <delayChannel prefix="/TEST2" delayTime="10000" batchTime="1500" batchSize="150" /> </delayManager> |
Configures an individual stream of delayed data.
Batching enables you to collect a number of updates and send them to peers with one command. When sending batches, your DataSource will keep all updates received in delayTime milliseconds, and then wait batchTime milliseconds before sending them.
A DelayChannel object is created and added to the DelayManager
for each instance of this tag.
Attributes:
| Name | Type | Default | Required | Description |
| prefix | string | "" | No | Object name prefix, to differentiate this channel's data from the real-time stream and other delayed data. |
| delayTime | long | 1000 | No | Milliseconds to delay data by. |
| batchTime | long | 500 | No | Milliseconds to delay batch of updates by. |
| batchSize | int | 10 | No | Starting point of number of updates in that batch. |
Example:
| <delayChannel prefix="/TEST1" delayTime="5000" batchTime="1000" batchSize="100" /> |
Parent tag containing the elements which configure how this application communicates with its peers. Multiple destinations should be used to enable failover. If multiple destinations are present then connections to the destinations will be attempted in the order the tags appear in the XML.
Attributes:
| Name | Type | Default | Required | Description |
| ssl | boolean | false | No | If true, uses the Secure Sockets Layer (SSL) protocol when connecting to another peer. |
| msgQueueSize | int | 200 | No | Sets the size of the message queue that will buffer outgoing messages in the event of the peer disconnect and then send them when the peer connection is re-established. |
| ejectExistingConnection | boolean | true | No | Whether this peer (if listening for inbound connections) should eject any existing connection when a new peer connection is received. |
| enableTCPNoDelay | boolean | false | No | Set whether the peer socket should use TCP_NODELAY. This setting should be true if the lowest possible message latency is required. When set to true, there is a trade-off of maximum throughput (messages / second) that will be possible. |
<peer> tags can contain the following elements:
| (either 0 or 1 occurrence) | |
|
<local> |
(either 0 or 1 occurrence) |
|
<remote> |
(either 0 or 1 occurrence) |
| (unlimited occurrences) |
Example:
<peer>
<local id="112" name="CaplinTest" type="ACTIVE CONTRIB"/>
<destination address="merlin" port="25007" />
<destination address="merlin2" port="25005" />
</peer> |
Configures the rate of heartbeats sent between peers which indicate the health of connections. This heartbeat rate is just a suggestion: on connection to a peer, the DataSources will negotiate a suitable heartbeat rate. If the peer doesn't support heartbeats then they will not be sent.
Attributes:
| Name | Type | Default | Required | Description |
| time | long | -1 | No | The time, in milliseconds, between heartbeats. The default (-1) means heartbeats are turned off. A suggested time is 15000 as this will send a heartbeat every 15 seconds if no other traffic is being sent, which is suitable for most firewall configurations. |
| slackTime | long | 5000 | No | The delay, in milliseconds, after the time attribute before the peer connection is terminated and your application attempts to reconnect due to no heartbeat being received. |
Example:
| <heartbeat time="5000" slackTime="2000" /> |
Configures the type information that the DataSource sends to this peer to identify its capabilities.
Attributes
| Name | Type | Default | Required | Description |
| name | string | appName (<dataSource> attribute) | No | Name of this DataSource. Overrides the appName attribute of the <dataSource> tag's attribute. |
| id | int | id (<dataSource> attribute) | No | ID number of this DataSource. Overrides the id attribute of the <dataSource> tag's attribute. |
| type | string | "" | No | Type of this DataSource. Possible values are shown below and can be either one or both, space separated. This type information is used by the peer so that it knows what operations this DataSource provides. Without either values the peer assumes this DataSource to be a Broadcast source that doesn't support Contributions.
|
Example:
| <local id="112" name="CaplinTest" type="ACTIVE CONTRIB"/> |
Configures the expected capabilites of this peer. N.B: once connected, this peer may send different capabilities, which will override those set here.
Attributes:
| Name | Type | Default | Required | Description |
| name | string | src_[index of peer] | No | Name of DataSource peer. Will be overridden by the peer once the connection is established. |
| id | int | 0 | No | ID number of the DataSource peer that will connect. |
| type | string | "" | No | Type of this DataSource. Possible values are shown below and can be either one or both, space separated. This type information is used by the DataSource so that it knows what operations this peer is expected to provide. This may differ from the actual operations that the peer provides, these will be set at connection time. Without either values this DataSource assumes the peer to be a Broadcast source that doesn't support Contributions.
|