Debugging DLL Loading Issues in Windows

Attempting to debug dynlib loading issues in Windows can be tricky – on Linux you can use strace to monitor dynamic linking and see what libraries are being loaded as the application runs. Having just had some issues with dynlib loading issues in an R extension I wrote, I was keen to find a good way to debugging the issues as they occurred. Thankfully, there are some good utilities out there for debugging this kind of thing:

Dependency Walker is an old SDK tool that is still invaluable. Drag a .DLL or .EXE into the application window and it will show you the dependencies, what type of loading mechanism they use (e.g. eager or delay load), and also what exported functions from the dependent DLLs are actually used by the .DLL or .EXE you have selected.

Here is a screenshot of depends.exe operating on my R extension DLL. Note that MSCVR80.DLL and R.DLL are both highlighted as missing – this is extremely useful for figuring out dependent DLL load issues.

Dependency Walker

Dependency Walker

The SysInternals Process Monitor is an incredibly useful low-level Swiss Army Knife utility that can be used, among other things, to monitor dynamic library loading activity as it occurs, using the file activity view. Here is a view of Process Monitor monitoring R as it loads my extension DLL. The subsequent dependent DLL loading activity is highlighted.

Process Explorer

Process Explorer

R/Reuters Real-Time Data Extension

Last week, I posted an R extension DLL that downloads historical data from Reuters using the SFC API. This time around, I am posting an extension DLL that can subscribe to real-time updates using the RFA API. This extension allows you to specify a list of items and data fields, and subscribe for a specified amount of time to updates on those items.

Here is an example:

# Load the DLL
dyn.load("RfaClient")

# Low-level subscription function wrapper
rsub < - function(time, items, func, sessionName, config, debug) {
.Call("subscribe", as.integer(time), items, func , sessionName, config, debug)
}

# Define items and fields to subscribe to
items <- list()
fields <- c("BID","ASK","TIMCOR")
items[[1]] <- c("IDN_SELECTFEED", "GBP=", fields)
items[[2]] <- c("IDN_SELECTFEED", "JPY=", fields)

# Callback function (invoked when data items update)
callback <- function(df) {
print(paste("Received an update for",df$ITEM, ", update time=", df$TIMCOR))
}

# Subscribe for 5 seconds using the supplied config parameters
rsub(5000, items, callback, "clientSession", "rfa.cfg", FALSE)

A short introductory guide is supplied:

Introduction (PDF)

Introduction (PDF)

The functionality is pretty basic right now, and there may be issues with the code (e.g. memory leaks, performance issues). Any feedback is gratefully received.

The package, including source, can be downloaded here:

rfaclient.zip

It was built with R 2.8.0 and RFA C++ 6.0.2, using Visual C++ 2005.

Working With Unicode Symbols in R and Vim

I came across a Vim feature today that I had forgotten about for a while – digraphs. Digraphs are basically Vim abbreviations for encoded character shortcuts. Typing <ctrl-K> followed by the digraph will insert the mapped character into the file. For instance, to insert the Euro symbol, type

<CTRL-K>Eu.

A list of existing digraphs can be shown using :digraph, and a new digraph can be added using the same command.

For instance, to add new digraphs for the Unicode male (♂) and female (♀) gender symbols, which are Unicode code points 9794/0×2642 and 9792/0×2640 respectively, you can type:

:dig Ma 9794
:dig Fe 9792

Similarly, R can insert Unicode code points into text and labels, and even data point symbols. One little-known convention in R is that if a data point symbol is negative, it is taken to be a Unicode code point. The following example generates some random data, and then displays the data with either a male or female symbols depending on whether the data point is divisible by 2 or not. Also, the symbols are inserted into the plot title using the \u character prefix.

> gender < - rbinom(n=100, size=100, prob=0.5)
> plot(gender, cex=2.5,
pch=ifelse(gender %% 2 == 0, -0x2642L, -0x2640L),
col=ifelse(gender %% 2 == 0, 2, 3), main="\u2640 and \u2642 Trials")

Plot Example

Plot Example

R/Reuters Time Series Data Extension

This is an upload of the historical data retrieval component of the R/Reuters interface that I presented at the R user conference in August 2008 (see the presentation here).

The extension is a C++ DLL that uses the Reuters SFC API to retrieve limited time series data from the TS1 time series service. It is compiled under Visual Studio 2005 and Windows XP, but could be cross-compiled to Linux without too much difficulty.

The DLL comes with a .R file that contains commands to load the DLL, and set up some necessary initialization. If you are familiar with Reuters infrastructure and APIs, the following points are important:

  • This extension has been built with support for SSL infrastructure only;
  • You will need to provide appropriate parameters to the init() function (see the documentation for details).

The extension, plus the associated source and documentation can be downloaded here:

reuters_ts1.zip

The package makes retrieving time series data for analysis easy: I have provided a single entrypoint for retrieving data, called fetch(). This will return a data frame with a Date column and one or more data columns, depending on the data being requested.

Some examples follow:

First, an example that fetches historical daily price data for Microsoft stock and creates a price/volume chart (the historical data returned contains OHLC price data and volume data):

msft.vol < - fetch("MSFT.O",sd="1/1/2008",ed="10/31/2008")
layout(matrix(c(1,1,1,1,2,2), 3, 2, T))
plot(msft.vol$Date, msft.vol$CLS, main="MSFT Close Price", type="l", ylab="", xlab="Date")
plot(msft.vol$Date, msft.vol$VOL, main="Volume", type='h', ylab="", xlab="")

MSFT Chart

MSFT Chart

The next example retrieves historical price data for the highest and lowest-rated ABX subprime indices – sometimes called the barometers of the subprime crisis:

abx.aaa < - fetch("ABXAA38FGB=MP", sd="1/1/2007", ed="12/1/2008", t="d")
abx.bbb <- fetch("ABXBM38FGB=MP", sd="1/1/2007", ed="12/1/2008", t="d")
op <- par(mfcol=c(2,1),las=2)
plot(...)

ABX AAA and BBB- Indices

ABX AAA and BBB- Indices

Finally, an example that shows the yield spreads between AAA and BBB-rated benchmark corporate bonds, and a 10-year US treasury bond:

library(zoo)
aaa10y < - fetch("AAAUSD10Y=", n=600, t="d")
bbb10y <- fetch("BBBUSD10Y=", n=600, t="d")
ust10y <- fetch("US10YT=RR", n=600, t="d")

aaa10y.series <- zoo(aaa10y, order.by=aaa10y$Date)
bbb10y.series <- zoo(bbb10y, order.by=bbb10y$Date)
ust10y.series < - zoo(ust10y, order.by=ust10y$Date)

full.series <- merge(aaa10y.series, bbb10y.series, ust10y.series, all=FALSE)
...

Yield Spreads

Yield Spreads

There are many more examples, plus documentation of the required R functions and their usage, in the manual:

Introductory Manual

Introductory Manual

UPDATE: The above package was built in Visual C++ 2005 using debug mode. I have attached another package below that was built in release mode. This package also includes the msvcrt redistributable DLLs, in case you get linker issues when R is loading the extension (as it was built using the MSVCRT 8.0 runtime).

reuters_ts_release1.zip

NB: This extension also depends on the Reuters SFC DLLs – specifically it loads ssl45w3280.dll and sipc3280.dll. These are available in the Reuters SSL C++ SDK version 4.5.5 (the latest version at the time of writing). I cannot legally redistribute these DLLs, so I would suggest that you download them from the Reuters development support site. Once downloaded, these DLLs should be in your system PATH (or in the same directory as the extension), so they can be loaded on demand.

DISCLAIMER: This software is not developed by Thomson Reuters and Thomson Reuters are in no way responsible for support or malfunction of this software.

The Credit Crunch in a TreeMap

FinViz is a site that aggregates a lot of financial information and presents it in interesting and innovative ways. The centrepiece of the site is a very impressive “squarified treemap” (see here for an interesting article on the history of the treemap as a data visualization tool). The map is apparently built using the Google Maps API, and is densely packed with information.

A couple of snapshots I found particularly interesting… here is the 3-month performance treemap for the S&P 500:

3-month S&P performance treemap

3-month S&P performance treemap

And here is the 1-week performance treemap:

1-week S&P performance treemap

1-week S&P performance treemap

It is a pretty interesting comparison – check out the tech sector hammering in the 3-month view (AAPL and GOOG particularly).

As an aside – has anyone else seen any interesting examples of treemaps or Voronoi maps (apart from the usual disc space explorers)?