Building QuickFIX on Solaris

I just had to build the QuickFIX C++ client library on Solaris, and here are a few notes that might be useful when doing the same:

* Make sure you have /usr/ccs/bin in your PATH (so that ar can be found);
* If you are using gmake, you may need to export MAKE=gmake before you run configure.
* If building static libs as well as .so files, run configure --enable-shared=yes.

Once the libraries are built, you can compile and link using:

$ g++ -I /usr/local/include/quickfix/include/ main.cpp libquickfix.a -o main -lxml2

This will statically compile against quickfix and dynamically link against libxml2.

The Elements of Modern Javascript Style

In the spirit of the last post, I wanted to share my newfound interest in another language that I previously hated. Ive always treated JavaScript like an unloved cousin. I hated dealing with it, and my brief encounters with it were really unenjoyable. However, I’ve had to deal with it a bit more recently, and thankfully my earlier encounters are distant memories, and the more recent experience is much better. There are two reasons for this: one is the excellent JQuery framework, and the other is the disparate collection of elegant patterns and best practises that have made JavaScript into a reasonably elegant language to work with.

The single best reference I have found so far that elucidates on all of these is the excellent “Javascript Web Applications” from O’Reilly, which expands on many facets of modern JavaScript usage. I would definitely recommend checking this book out.

The Elements of Modern C++ Style

Herb Sutter has a fantastic short article over at his site on the new C++11 standard. The new extensions to the language are way overdue. One particular example that I really liked was the application of library algorithms and lambdas to create “extensions” to the language. For instance, the example below adds a “lock” construct (similar to the one in C#):

[sourcecode language=”cpp” wraplines=”true”]
// C#
lock( mut_x ) {
… use x …
}

// C++11 without lambdas: already nice, and more flexible (e.g., can use timeouts, other options)
{
lock_guard<mutex> hold { mut_x };
… use x …
}

// C++11 with lambdas, and a helper algorithm: C# syntax in C++
// Algorithm: template<typename T, typename F> void lock( T& t, F f ) { lock_guard<t> hold(t); f(); }
lock( mut_x, [&]{
… use x …
});
[/sourcecode]

Of course, all of this is dependent on compiler support. A couple of nice resources on the current state of C++11 support are:

http://www.aristeia.com/C++11/C++11FeatureAvailability.htm

and

https://wiki.apache.org/stdcxx/C%2B%2B0xCompilerSupport

Deducing the JDK Version of a .jar File

Here is a little script that uses Pyton to examine the contents of a jar file (or specifically the first .class file it comes across) and then reads the major version byte and maps it to a JDk version. May be useful if you have a bunch of jars compiled by different JDKs and want to figure out which is which.

[sourcecode language=”python”]#!/usr/bin/python

import zipfile
import sys
import re

class_pattern=re.compile("/?\w*.class$")

for arg in sys.argv[1:]:
print ‘%s:’ % arg,
file=zipfile.ZipFile(arg, "r")

for entry in file.filelist:
if class_pattern.search(entry.filename):
bytes = file.read(entry.filename)
maj_version=ord(bytes[7])
if maj_version==45:
print "JDK 1.1"
elif maj_version==46:
print "JDK 1.2"
elif maj_version==47:
print "JDK 1.3"
elif maj_version==48:
print "JDK 1.4"
elif maj_version==49:
print "JDK 5.0"
elif maj_version==50:
print "JDK 6.0"
break
[/sourcecode]

Gdb Macros for R

When debugging R interactively, one hurdle to navigate is unwrapping SEXP objects to get at the inner data. Gdb has some useful macro functionality that allows you to wrap useful command sequences in reusable chunks. I recently put together the following macro that attempts to extract and print some useful info from a SEXP object.

It can be used as follows. For instance, given a SEXP called “e”:

(gdb) dumpsxp e
Type: LANGSXP (6)
Function:Type: SYMSXP (1)
"< -" Args:Type: LISTSXP (2) (SYMSXP,LISTSXP)

We can see that e is a LANGSXP, and the operator is “< -". Functions have different components - here we can see the function representation (the SYMSXP) and the function arguments (the LISTSXP).

Some knowledge of LANGSXP structure is useful here. For instance, if we know that for a LANGSXP that CAR(x) gives us the function and CDR(x) gives us the arguments, we can view the components individually.

To see the first component:

(gdb) dumpsxp CAR(e)
Type: SYMSXP (1)
"< -"

The arguments are given by the CDR of e. We can then crack open the list and view the function arguments, recursively looking through the pairlist until we get to the end:

(gdb) dumpsxp CDR(e)
Type: LISTSXP (2)
(SYMSXP,LISTSXP)
(gdb) dumpsxp CADR(e)
Type: SYMSXP (1)
"x"
(gdb) dumpsxp CADDR(e)
Type: LANGSXP (6)
Function:Type: SYMSXP (1)
"sin"
Args:Type: LISTSXP (2)
(REALSXP,NILSXP)
(gdb) dumpsxp CADDDR(e)
Type: NILSXP (0)

The NILSXP tells us that we’ve got to the end of the list.

The GDB macro is below. Put it in your .gdbinit to automatically load it when gdb starts up.

[sourcecode language=”c”]
define dumpsxp
if $arg0==0
printf "uninitialized variable\n"
return
end

set $sexptype=TYPEOF($arg0)

# Typename
printf "Type: %s (%d)\n", typename($arg0), $sexptype

# SYMSXP
if $sexptype==1
# CHAR(PRINTNAME(x))
print_char PRINTNAME($arg0)
end

# LISTSXP
if $sexptype==2
printf "(%s,%s)\n", typename(CAR($arg0)), typename(CDR($arg0))
end

# CLOSXP
if $sexptype==3
dumpsxp BODY($arg0)
end

# PROMSXP
# Promises contain pointers to value, expr and env
# tmp = eval(tmp, rho);
if $sexptype==5
printf "Promise under evaluation: %d\n", PRSEEN($arg0)
printf "Expression: "
dumpsxp ($arg0)->u.promsxp.expr
# Expression: (CAR(chain))->u.promsxp.expr
end

# LANGSXP
if $sexptype==6
printf "Function:"
dumpsxp CAR($arg0)
printf "Args:"
dumpsxp CDR($arg0)
end

# SPECIALSXP
if $sexptype==7
printf "Special function: %s\n", R_FunTab[($arg0)->u.primsxp.offset].name
end

# BUILTINSXP
if $sexptype==8
printf "Function: %s\n", R_FunTab[($arg0)->u.primsxp.offset].name
end

# CHARSXP
if $sexptype==9
printf "length=%d\n", ((VECTOR_SEXPREC)(*$arg0))->vecsxp.length
#print_veclen $arg0
print_char $arg0
end

# LGLSXP
if $sexptype==10
set $lgl=*LOGICAL($arg0)
if $lgl > 0
printf "TRUE\n"
end
if $lgl == 0
printf "FALSE\n"
end
end

# INTSXP
if $sexptype==13
printf "%d\n", *(INTEGER($arg0))
end

# REALSXP
if $sexptype==14
print_veclen $arg0
print_double $arg0
end

# STRSXP
if $sexptype==16
print_veclen $arg0
set $i=LENGTH($arg0)
set $count=0
while ($count < $i)
printf "Element #%d:\n", $count
dumpsxp STRING_ELT($arg0,$count)
set $count = $count + 1
end
end

# VECSXP
if $sexptype==19
print_veclen $arg0
end

# RAWSXP
if $sexptype==24
print_veclen $arg0
end

end

define print_veclen
printf "Vector length=%d\n", LENGTH($arg0)
end

define print_char
# this may be a bit dodgy, as I am not using the aligned union
printf "\"%s\"\n", (const char*)((VECTOR_SEXPREC *) ($arg0)+1)
end

define print_double
printf "%g\n", (double*)((VECTOR_SEXPREC *) ($arg0)+1)
end
[/sourcecode]

Custom Checkstyle Rule Example

Here is an example of a custom Checkstyle rule that catches the following situation: sometimes an Exception can be caught and an error message logged, but the underlying exception (and thus the stack trace) may not be logged at all. This is not a problem generally, unless the exception tends wrap an underlying cause (i.e. one or more nested exceptions). The rule is designed to catch instances like:

[sourcecode language=”java”]
try {}
catch (Exception e) {
logger.error("blah");
}
[/sourcecode]

With the assumption that the following is better:

[sourcecode language=”java”]
try {}
catch (Exception e) {
logger.error("blah", e);
}
[/sourcecode]

Here is the code for the custom check:

[sourcecode language=”java”]
package au.com.national.efx.build;
import java.util.ArrayList;
import java.util.List;

import com.puppycrawl.tools.checkstyle.api.Check;
import com.puppycrawl.tools.checkstyle.api.DetailAST;
import com.puppycrawl.tools.checkstyle.api.TokenTypes;

/**
* Check that attempts to catch instances of the following:
* <code>
* catch (Exception e) { logger.error("foo"); }
* </code>
*
* with the assumption that the following is preferable:
*
* <code>
* catch (Exception e) { logger.error("foo", e); }
* </code>
* @author rwinston
*
*/
public class SwallowedExceptionInLoggerCheck extends Check {

@Override
public int[] getDefaultTokens() {
return new int[] { TokenTypes.LITERAL_CATCH };
}

/**
* Get ident of exception
* Try to find it in logger error/warn parameter list
*/
@Override
public void visitToken(DetailAST aAST) {
super.visitToken(aAST);

final DetailAST parameterDef = aAST.findFirstToken(TokenTypes.PARAMETER_DEF);
final DetailAST ident = parameterDef.findFirstToken(TokenTypes.IDENT);
final String exceptionIdent = ident.getText();

final DetailAST slist = aAST.findFirstToken(TokenTypes.SLIST); // Find ‘{‘
// Find all method calls within catch block
final List<DetailAST> variables = findChildASTsOfType(slist, TokenTypes.METHOD_CALL);

try {
for (DetailAST methodCall : variables) {
DetailAST dot = methodCall.findFirstToken(TokenTypes.DOT);

// I’m assuming the last child will be the method name called
DetailAST lastIdent = dot.getLastChild();

if (lastIdent.getText().equals("error")) {
// Ok, now check that the ELIST contains an IDENT matching
// the exception name
DetailAST elist = methodCall.findFirstToken(TokenTypes.ELIST);
boolean exceptionInParameterList = false;
for (DetailAST identAST : findChildASTsOfType(elist, TokenTypes.IDENT)) {
if (identAST.getText().equals(exceptionIdent))
exceptionInParameterList = true;
}

if (!exceptionInParameterList) {
log(methodCall, "error() method does not contain caught Exception as a parameter");
}
}
}
} catch (Exception e) { e.printStackTrace(); }

}

/**
* Recursively traverse an expression tree and return all
* ASTs matching a specific token type
* @param parent
* @param type
* @return
*/
private List<DetailAST> findChildASTsOfType(DetailAST parent, int type) {
List<DetailAST> children = new ArrayList<DetailAST>();

DetailAST child = parent.getFirstChild();
while (child != null) {
if (child.getType() == type)
children.add(child);
else {
children.addAll(findChildASTsOfType(child, type));
}
child = child.getNextSibling();
}
return children;
}

}

[/sourcecode]

Project Euler Problem #21

This is a solution for problem 21 on the Project Euler website. It consists of finding the sum of all the amicable numbers under 10000. This was pretty easy to solve, but the solution could probably be improved quite a bit.

Solution #1 in R is as follows (it calculates the proper divisors of each number using prop.divs, and then adds up the sequence of amicable numbers in the main function).

[sourcecode language=”matlab”]
prop.divs <- function(x) {
if (x == 1) return (1)
divs <- integer(30)
j <- 1
divs[j] <- 1
j <- j + 1
for (i in 2:(floor(x/2))) {
if ((x %% i) == 0) {
divs[j] <- i
j <- j + 1
}
}
sum(divs[1:(j-1)])
}

problem.21 <- function(N) {
s <- 0
for (i in 2:N) {
da <- prop.divs(i)
if (da == i) next
db <- prop.divs(da)
if ( db==i ) {
s <- s + da + db
}
}
s/2
}
[/sourcecode]

The s/2 is needed as each factor is added twice during the calculation.

This gives the correct answer, but the implementation is a bit naive. I remembered coming across an article about prime factors and proper divisors on PlanetMath a while ago, and this seemed like potentially a more efficient way to calculate the factors involved. Specifically, the sum of proper divisors of a number n can be given by:

\[ \prod_{i=1}^k\frac{p_i^{m_i+1}-1}{p_i – 1}-\prod_{i=1}^kp_i^{m_i} \]

The second attempt at this problem looked like the following:

[sourcecode language=”r”]
prime.sieve <- function(n) {
a <- seq.int(1,n)
p <- 1
M <- as.integer(sqrt(n))
while ((p <- p + 1) <= M) {
if (a[p] != 0)
a[seq.int(p*p, n, p)] <- 0
}
a[a>1 & a>0]
}

sum.proper.divisors <- function(x) {
primes <- prime.sieve( x )
primes <- primes[ x %% primes == 0]

geo.sum <- numeric(length(primes))
i <- 1

for (prime in primes) {
n <- x
curr <- 0
while (n %% prime == 0) {
curr <- curr + 1
n <- n %/% prime
}
geo.sum[i] <- ( (prime^(curr+1) – 1)/(prime – 1) )
i <- i + 1
}
prod(geo.sum)-x
}

problem.21_2 <- function(N) {
s <- 0
for (i in 2:N) {
da <- sum.proper.divisors(i)
if (da == i) next
db <- sum.proper.divisors(da)
if (db==i) s <- s + da +db
}
s/2
}
[/sourcecode]

This also gives the correct answer, but with much reduced runtime overhead:


> system.time(problem.21(10000))
user system elapsed
103.943 0.511 106.978
> system.time(problem.21_2(10000))
user system elapsed
24.834 0.160 26.565

Scripting Maven Deployments with Ruby

Recently I had to import a number of legacy projects into a Maven-type structure. I knocked up the following script to make the task easier for repeated applications. Basically what it does is the following:

  • Tried to parse the filenames of jars in the current directory;
  • Does a Nexus search to see if it can locate the artifact;
  • If it finds the artifact in Nexus, it can use the GAV parameters for that artifact, otherwise, it generates a GAV stanza for inclusion in the POM.
  • For each artifact that could not be located in Nexus, it generates a mvn deploy:deploy-file command to upload the dependency

Error handling is non-existent in this file, so caveat emptor!

[sourcecode language=”ruby” wraplines=”true”]
#!/usr/bin/ruby

require ‘net/http’
require ‘rexml/document’
include REXML

# Nexus server
host="nexus-server"
port=8080

# Could also use GAV parameters in query string
url="http://#{host}:#{port}/nexus/service/local/data_index?q="

print "<dependencies>","\n"

# Open file for dependencies entries
depsFile=File.open("dependencies.xml",’w’)
depsFile.write("</dependencies><dependencies>\n")

# Track unprocessed jars
errors=[]

# Jars to be uploaded to Nexus
uploads=[]

for jarFile in Dir.glob("*.jar")
if jarFile =~ /([A-Za-z\-\.0-9]+?)(?:\-)?(\d+(?:\.\d+)*)?.jar/
artifact=$1
version=$2

if artifact =~ /(\S+)\.(.*)/
group=$1
artifact=$2
else
group=artifact.gsub("\-", "\.")
end

if version.nil?
version="1.0"
end

# Create a default stanza template
stanza = "<dependency>\n"
stanza < < ‘ <groupId>#{group}\n’
stanza < < ‘ <artifactId>#{artifact}<artifactid>\n’
stanza < < ‘ <version>#{version}\n’
stanza < < " <packaging>jar\n"
stanza < < "</dependency>\n\n"

puts "Searching Nexus for #{artifact}"
query="#{url}#{artifact}"

resp = Net::HTTP.get_response(URI.parse(query))
respDoc = REXML::Document.new(resp.body)
XPath.each(respDoc, "//search-results/totalCount") do |count|
matches = Integer(count.text)
if matches > 0

idx=1; artifactMap = {}
XPath.each( count, "//data/artifact") do |ar|
artifactId=ar.elements["artifactId"].text
artifactGroupId=ar.elements["groupId"].text
artifactVersion=ar.elements["version"].text
artifactRepo=ar.elements["repoId"].text
puts "#{idx}. #{artifactGroupId}:#{artifactId}:#{artifactVersion} (repo:#{artifactRepo})"
artifactMap[idx]={"group",artifactGroupId, "artifact",artifactId, "version",artifactVersion}
idx+=1
end

puts "Found #{matches} possible match(es) for #{group}:#{artifact}:#{version}"
puts "Enter artifact number, or enter to use generated artifact:"
num=gets.chomp

if !num.empty?
num=Integer(num)
entry=artifactMap[num]
group=entry["group"];artifact=entry["artifact"];version=entry["version"]
else
uploads.push( {"artifact",artifact,"group",group,"version",version,"file",jarFile} )
end

else
uploads.push( {"artifact",artifact,"group",group,"version",version,"file",jarFile} )
end
stanza = eval(‘"’ + stanza + ‘"’)
depsFile.write(stanza)
end
else
errors < < jarFile
end
end

depsFile.write("</dependencies>\n")
depsFile.close

puts "Finished processing.\nCheck the file dependencies.xml for generated XML"

# Generate the upload script
File.open("upload_deps.sh", "w") { |f|
f.puts "#!/bin/bash\n\n"
uploads.each do |dep|
# Send the file to the correct repo
repo="http://#{host}:#{port}/nexus/content/repositories/"
if dep["version"] =~ /(.*)SNAPSHOT/
repo < < "snapshots"
else
repo << "releases"
end

line="mvn deploy:deploy-file -Dfile=#{dep["file"]} -DartifactId=#{dep["artifact"]} -DgroupId=#{dep["group"]} -Dversion=#{dep["version"]} -Dpackaging=jar -DrepositoryId=nexus -DgeneratePom=true -Durl=#{repo}"

f.puts "echo \"Uploading #{dep["file"]}…\""
f.puts line
end
}

puts <<HERE
The file upload_deps.sh contains Maven commands to upload dependencies not currently in Nexus.

This assumes that you have a stanza in $MVN_HOME/conf/settings.xml as follows:

<server>
<id>nexus</id>
<username>$USERNAME</username>
<password>$PASSWORD</password>

Where the appropriate Nexus deployment credentials are in <username> and <password>.
HERE

if !errors.empty?
puts "The following jar files could not be processed and need to be manually defined:"
errors.each do |error|
puts error
end
end

[/sourcecode]

GSoC Commons-Net SSH Concluded

Today is officially “pencils-down” day for the Google Summer of Code Project 2009. I have been a mentor this year for the Apache Commons-Net SSH project, which aims to add SSH and SCP support to Commons-Net.

The project has been a great success, mainly down to the super work performed by the student, Shikhar, who has put in tremendous work to get a fully functional SSH/SCP client built (with thanks to the efforts of the Mina SSHD project, whose codebase we originally based the effort on). All of the goals for the project have been ticked, and some extra ones accomplished too.

I had great help and input from Chico, another Apache committer, throughout the project, and so it’s been a great experience all round. This will form the basis of a release, but for now, the code is hosted on googlecode at: http://code.google.com/p/commons-net-ssh/