On one server I have an xmpp chat system (ejabberd running on debian) running over SSL. On other servers I have instances of Apache Tomcat running various web applications.

• I created a new java.util.logging.Handler that could be used with tomcat.
• Created xmpp user accounts within ejabberd for each webapp I wanted to monitor.
• Updated the logging.properties file for each webapp and restarted.

The handler was setup to auto-accept contact requests (not a problem on an internal chat server), so I added the new xmpp user accounts as contacts on my own account. Step 1 successful, I can now see when the webapps are online and do something about it when I notice they go offline. To receive live logging information, I’d also built the following commands into the handler.

• !subscribe
• !unsubscribe
• !level <LEVEL_NAME>

First two are pretty obvious, they allow you to subscribe and unsubscribe from being sent log messages via chat. The third command allows you to select the minimum level messages you want to see – on production servers I get way too many messages if I listen to ALL, so I tend to choose WARNING as I’m only interested in seeing if things go wrong.

An example logging.properties file updated from the default one that ships with tomcat;

handlers = org.apache.juli.FileHandler, uk.org.tilion.logging.XmppHandler, java.util.logging.ConsoleHandler

uk.org.tilion.logging.XmppHandler.server = chat.tilion.org.uk
uk.org.tilion.logging.XmppHandler.port = 5222
uk.org.tilion.logging.XmppHandler.username = webapp1-name
uk.org.tilion.logging.XmppHandler.password = webapp1-password

org.apache.juli.FileHandler.level = ALL
org.apache.juli.FileHandler.directory = ${catalina.base}/logs

java.util.logging.ConsoleHandler.level = SEVERE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

The code is squeezed into a single file and packaged in a maven project that uses smack/smackx to perform the xmpp communication. I’ve not attached it to this post, but if anyone is interested, leave a comment below and I’ll make it available.

If I get time I’ll create a version for protocol 68 that can decode the Huffman compressed packet data on the fly, but don’t hold your breath!

Instructions can be found in the code comments, otherwise leave me a message below.

package uk.org.tilion.quake3.proxy;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.channels.DatagramChannel;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.charset.Charset;
import java.util.Iterator;

/**
 * A Quake3 proxy server that communicates using protocol 43, 1.11 - 1.16 point
 * release of quake3 engine. It's designed to handle a single client and provide
 * a way of experimenting with the quake3 protocol on the fly.
 * 
 * Packet fragmentation is ignored for simplicity, with sequenceId being read
 * without performing fragmentation checks.
 * 
 * Example handshake (OOB prefixed packets)
 *   CLIENT : getChallenge
 *   SERVER : challengeResponse <ID>
 *   CLIENT : connect "cg_predictItems1sexmalehandicap100color3snaps40rate10000modeldoom/rednameUnnamaedPlayerprotocol68qport<PORT>challenge<ID>"
 *   SERVER : connectResponse
 * 
 * Client game data packet
 * +-----------------+-----+----------+-----------+
 * | NAME            | LEN | ENCODING | ENDIANESS |
 * +-----------------+-----+----------+-----------+
 * |  SequenceNumber |  32 |     None |    Little |
 * |           QPort |  16 |     None |    Little |
 * |                 |     |      XOR |           |
 * 
 * Server game data packet
 * +-----------------+-----+----------+-----------+
 * | NAME            | LEN | ENCODING | ENDIANESS |
 * +-----------------+-----+----------+-----------+
 * |  SequenceNumber |  32 |     None |    Little |
 * |                 |     |      XOR |           |
 * 
 * If anyone gets time to implement the XOR decoding, do let me know! I suspect
 * it's the Netchan_UnScamblePacket() function from qcommon/net_chan.c in the 
 * 1.32 full source release.
 * 
 * @author Darren Edmonds
 */
public class Protocol43ProxyServer {

    private static final int BUFFER_SIZE = 1024 * 2;  // size in bytes
    
    private int localPort;
    private SocketAddress serverAddr;
    private SocketAddress clientAddr;
    private DatagramChannel serverChannel; // proxy <==> quake3 server
    private DatagramChannel clientChannel; // proxy <==> quake3 client
    private boolean running;
    
    Protocol43ProxyServer(int localPort, String remoteServer, int remotePort) {
        this.localPort = localPort;
        this.serverAddr = new InetSocketAddress(remoteServer, remotePort);
    }

    /**
     * Start the proxy server
     * @throws IOException 
     */
    public void start() throws IOException {
        Selector selector = Selector.open();
        
        this.clientChannel = DatagramChannel.open();
        this.clientChannel.configureBlocking(false);
        this.clientChannel.socket().bind(new InetSocketAddress(this.localPort));
        this.clientChannel.register(selector, SelectionKey.OP_READ);
        
        this.serverChannel = DatagramChannel.open();
        this.serverChannel.configureBlocking(false);
        this.serverChannel.socket().bind(null);
        this.serverChannel.register(selector, SelectionKey.OP_READ);
        
        ByteBuffer buffer = ByteBuffer.allocateDirect(BUFFER_SIZE);
        buffer.order(ByteOrder.BIG_ENDIAN);

        this.running = true;
        Iterator<SelectionKey> it = null;
        SelectionKey key = null;
        while (this.running) {
            int n = selector.select();
            if (n == 0) continue; // nothing to do
            
            it = selector.selectedKeys().iterator();
            while (it.hasNext()) {
                key = it.next();
                
                if (key.isReadable()) {
                    this.handlePacket((DatagramChannel)key.channel(), buffer);
                }
                
                it.remove();
            }
        }
    }

    /**
     * Read a packet of data from the client (or server) datagram channel then
     * proxy it over to the other channel.
     * @param channel
     * @param buffer
     * @throws IOException 
     */
    private void handlePacket(DatagramChannel channel, ByteBuffer buffer)
            throws IOException {
        buffer.clear();
        
        if (channel == this.clientChannel) {
            SocketAddress sender;
            
            while ((sender = channel.receive(buffer)) != null) {
                if (this.clientAddr == null) this.clientAddr = sender;
                //System.out.println("CLIENT sent " + buffer.position() + " bytes");
                inspectClientPacket(buffer);
                writePacket(this.serverChannel, buffer, this.serverAddr);
            }
            
        } else {
            while (channel.receive(buffer) != null) {
                //System.out.println("SERVER sent " + buffer.position() + " bytes");
                inspectServerPacket(buffer);
                writePacket(this.clientChannel, buffer, this.clientAddr);
            }
        }
    }

    /**
     * Write buffer to channel, transmitting content to recipient
     * @param channel
     * @param buffer
     * @param recipient
     * @throws IOException 
     */
    private void writePacket(DatagramChannel channel, ByteBuffer buffer,
            SocketAddress recipient) throws IOException {
        buffer.flip();
        while (buffer.hasRemaining()) {
            channel.send(buffer, recipient);
        }
    }

    /**
     * Chance to inspect/modify packet sent from client before it is relayed
     * to the server
     * @param buffer
     */
    private void inspectClientPacket(ByteBuffer buffer) {
        buffer.order(ByteOrder.LITTLE_ENDIAN);
        int sequenceId = buffer.getInt(0);

        if (sequenceId == -1) { // 4 OOB bytes in the header
            byte[] textArr = new byte[buffer.position() - 4];
            int oldPos = buffer.position();
            buffer.position(4);
            buffer.get(textArr);
            buffer.position(oldPos);

            String command = new String(textArr, Charset.forName("UTF-8"));
            System.out.println("CLIENT " + command);
            
            if (command.startsWith("connect ")) {
                /* rewrite the qport value to reflect the proxy local port
                 * rather than the client local port - not critical for testing
                 * but required for multiple clients on same IP via NAT */
            }
        } else {
            int qport = buffer.getShort(4);
            System.out.println("CLIENT seq=  " + String.format("%15d", sequenceId));
            System.out.println("CLIENT qport=" + String.format("%15d", qport));
        }
        
        buffer.order(ByteOrder.BIG_ENDIAN);
    }

    /*
     * Chance to inspect/modify packet sent from server before it is relayed
     * to the client
     * @param buffer
     */
    private void inspectServerPacket(ByteBuffer buffer) {
        buffer.order(ByteOrder.LITTLE_ENDIAN);
        int sequenceId = buffer.getInt(0);
        
        if (sequenceId == -1) { // 4 OOB bytes in the header
            byte[] textArr = new byte[buffer.position() - 4];
            int oldPos = buffer.position();
            buffer.position(4);
            buffer.get(textArr);
            buffer.position(oldPos);

            String command = new String(textArr, Charset.forName("UTF-8"));
            System.out.println("SERVER " + command);
            
            if (command.startsWith("connectResponse")) {
                /* server puts client into connecting state and starts sending
                 * game updates */
            }
        } else {
            System.out.println("SERVER seq=  " + String.format("%15d", sequenceId));
        }
        
        buffer.order(ByteOrder.BIG_ENDIAN);
    }

    /**
     * Main
     * @param args 
     */
    public static void main(String[] args) {
        Protocol43ProxyServer server = new Protocol43ProxyServer(
                27960, 
                "myserver.com", // CHANGE TO REAL SERVER IP
                27961); // CHNAGE TO REAL SERVER PORT
        try {
            server.start();
        } catch (Exception e) {
            e.printStackTrace(System.err);
        }
        
        /* now start quake3.exe and connect localhost:27960
         * your connection will end up at myserver.com:27961, proxied via
         * localhost to allow you to inspect packets on the fly */
    }
}

The problem isn’t getting my signed certificate in formats that both Apache Tomcat and Apache HTTP will understand, that bit’s easy. The problem is that my certificate request was created using a private key stored in a java keystore, which Apache HTTP doesn’t understand. Quick google later and the solution wasn’t very difficult, here’s what I did!

GoDaddy created my certificate and provide their root CA bundle, let’s call them wild.tilion.org.uk.crt and gd_bundle.crt respectively.

I created the initial signing request with a java keystore, tomcat.keystore, under alias tilion.

Usage instructions for ExportPriv were quick and easy to follow.

javac ExportPriv.java Base64Coder.java
java ExportPriv tomcat.keystore tilion <password> | openssl pkcs8 -inform PEM -nocrypt > wild.tilion.org.uk.key

Apache2 configuration parameters:

SSLEngine on
SSLCertificateFile /path/to/wild.tilion.org.uk.crt
SSLCertificateKeyFile /path/to/wild.tilion.org.uk.key
SSLCertificateChainFile /path/to/gd_bundle.crt

Create and Import

I prefer to create plain file backed repositories as they don’t suffer the same inconsistency quirks as the berkley database backed repositories. The following will create a new repository called project in the current working directory.

svnadmin create project --fs-type fsfs

If you don’t have any files to import you can start creating files as you develop and then use svn add to put them into version management.

Checkout

The following will checkout the project into a directory called myproject, created in the current working directory.

svn checkout svn+ssh:/user@svn.domain.com/project/trunk myproject

Ignoring File/Directories

Build directories or log directories can be annoying when they continue to show up in the svn status command. You can instruct subversion to ignore them using

svn propedit svn:ignore target

Tagging and Branching

Make sure the tags or branches directory is created first.

svn mkdir svn+ssh://user@svn.domain.com/project/tags

From the base directory of your project, tag the release or create the branch.

svn copy . svn+ssh://user@svn.domain.com/project/tags/1_0_0

Updating a tagged revision

If you need to make updates to a tagged revision (say 2.1.0) when the main trunk has already progressed with new development (2.2.0 for instance) you can do it as follows.

Copy the 2.1.0 tag into a branch

svn copy svn+ssh://user@svn.domain.com/project/tags/2_1_0 svn+ssh://user@svn.domain.com/project/branches/2_1_1

Checkout this new branch and make code updates within it. You can commit changes as you would if you were working on the trunk.

svn checkout svn+ssh:/user@svn.domain.com/project/branches/2_1_1 myproject-2_1_1

When finished, tag the new version.

svn move svn+ssh:/user@svn.domain.com/project/branches/2_1_1 svn+ssh:/user@svn.domain.com/project/tags/2_1_1

Conflict Resolution

If you try to apply changes and end up with a file in conflict (marked with a C next to its name) you have 3 ways to solve this; merge text in the file by hand, copy one of the temporary files over the top of the original, run svn revert on the file.

If you fix the problem by hand, or by copying one of the temporary files over the top, you must let subversion know by using the following command.

svn resolved <FILE>

Merging changes from a tag/branch back into trunk

You need checked out copies of the tag/branch and trunk. It’s easier if changes in trunk are committed as it makes rolling back easier (just use the revert command).

Next, work out which revisions from the tag/branch you want to apply into trunk. You can do this with svn log (within the tag/branch) to see when it was created and what has been committed. From with trunk, run the following command.

svn merge -r 200:204 svn+ssh:/user@svn.domain.com/project/branch/big_change

At this point changes are made locally in trunk (201,202,203,204, 200 is not inclusive). Compile code, check changes work then commit them being very specific of what just happened in the log message, e.g. Merged branch/big_change r200:204 into trunk

Relocating a subversion repository

Maybe the server breaks down, or you just need to move the location of your subversion server. In theory you can do it with the following command.

svn switch --relocate svn+ssh:/user@svn.domain.com/project/branch/2_2_2 svn+ssh:/newuser@svn.newdomain.com/project/branch/2_2_2 .

The command above should recurse into subdirectories, although when I tried it, it didn’t work for me. Resorting back to sed you can do the same as follows.

for file in `find . -name 'entries'`; do sed s/user@svn.domain.com/newuser@svn.newdomain.com/g $file > $file'A'; done
for file in `find . -name 'entriesA'`; do mv `dirname $file`/entriesA `dirname $file`/entries; done

Using captivate I created a set of multiple choice questions – I won’t go into all the details of how to do this as it was the easy bit! Export the package as a SCORM 1.2 zip file.

Running the output from captivate straight through the ADL test suite you’ll see it fails with an initial error of ERROR: LMS Not initialized. If you look at the content window opened up by the test suite you’ll also notice there are javascript errors, the first one being access denied to scorm_support.js

The details of this fix have been borrowed, then embellished, from http://www.mylearning.be/2009/09/adobe-captivate-and-the-adl-scorm-test-suite-1-2-7/

To make sure that your SCO gets launched in the test suite, you need to edit the html file that is generated by Captivate when publishing your content (the .html that has the same name as your project .swf). Open the file with a text editor (Notepad), on the second line you will find <!– saved from url=(0013)about:internet –>. Delete that line.

Restart your test, and your SCO will now launch. But you will get errors in your test now. Fix number two: change the security settings of the Flash player on your machine.

• Get some Flash content playing in your browser. Any Flash animation will do. Go e.g. to www.adobe.com.
• Right-click on the animation, you will get the Flash context menu. Select Settings.
• You will get a little menu like this:
  
• Click the Advanced button. This will bring you to an Adobe Web site.
• In the table of contents on the left, click Global Security Settings Panel. This will show you a panel like this:
  
• Add the location where your ADL TestSuite software is installed to the trusted locations. The location of the TEST SUITE software, not the location of your zip file or your content files. Those get copied automatically to a TestSuite subfolder when you run the test.
• Close all your browser windows and re-run the test.

From here, I continue the explanation with my own experience … having unzipped the original zip file to make changes to the HTML file I decided to test with it unzipped as the test suite has that option – do not do this. I wasted a great deal of time trying to work out why LMSInitialize() was not being called by my content, to the extent that I actually hardcoded a change in to make sure it was called. This unzipped package now passed the test suite, but having never run anything through the test suite before I couldn’t tell if it was working correctly.

Everything now works, hurrah, so I rezip the package and decide to give it a final test. LMSInitialize() now gets called twice, hence the waste of all my effort – I then removed my hardcoded call. Now, when you run this package through the test suite, not only does it pass, but it shows you all the data communicated from the content package to the LMS (getValue/setValue calls).

In summary, the only thing wrong with the initial Captivate output is that it contains a comment (on line 2) that needs to be removed. I suspect the other fix is only needed to satisfy the test suite running the package from the local disk, as opposed to being served through a web server over HTTP.
]]> http://www.tilion.org.uk/2012/03/scorm-adobe-captivate-and-adl-test-suite-1-2-7/feed/ 1 http://www.tilion.org.uk/2012/03/tomcat-ssl-certificate-alias-tomcat-name-does-not-identify-a-key-entry/ http://www.tilion.org.uk/2012/03/tomcat-ssl-certificate-alias-tomcat-name-does-not-identify-a-key-entry/#comments Thu, 15 Mar 2012 11:28:55 +0000 Darren http://www.tilion.org.uk/?p=108 Continue reading →]]> A post that may help someone out if they get into the same situation I did with regards to importing SSL certificates into a java keystore for Tomcat.

When renewing my certificate, my CA had the ability to use my old CSR (certificate signing request) which I accepted as it saved me a few minutes. Before, I’d always started with an empty keystore, generated my private key, CSR, then imported my new certificate along with the any needed to complete the chain. It seemed easy, I just needed to import my new certificate into the old keystore, right?

keytool -import -alias intermed -keystore tomcat.keystore -trustcacerts -file gd_intermediate.crt
Enter keystore password:  
keytool error: java.lang.Exception: Certificate not imported, alias  already exists

I see, I’ve already got a certificate under that alias, so I need to remove it first. Into the manual, -delete option looks good and away we go … I delete and then import 2 certificates that make up the chain and do the same with my newly issued certificate. Update my tomcat config to be greeted by the following:

LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.io.IOException: Alias name tomcat does not identify a key entry

To cut a long story short, when you use the -delete option of keytool on an alias with a private key in it, it doesn’t just remove the certificate, it removes your private key as well. Adding in my new certificate is all well and good if I no longer have a private key associated with it! The correct thing to do is not use the -delete option at all, because keytool will not complain if you’re importing a new certificate like that over the top of an old one, e.g I already have a certificate in the alias ‘tomcat’ but …

keytool -import -alias tomcat -keystore tomcat.keystore -trustcacerts -file mydomain.com.crt
Enter keystore password:  
Certificate was added to keystore

What follows isn’t a new idea and doesn’t necessarily include new concepts. However, it is my take on a solution that allows me to continually extend if necessary – it’s often very difficult to extend someone else’s work, especially if it is closed source.

Screenshots

Players overview: when you announce a competition you’ll get a list of players stating their interest, some even paying you money, but at this point there is no guarantee they’ll turn up on the day.

Competitions overview: most competitions I’ve been involved in include a main event and then a plate event to keep early round losers interested. Competitions have multiple rounds and can be run simultaneously over the available tables.

Teams overview: for most competitions this tab would be better named as Registered Players, but the software is designed to work for single, or team events, in the same way. When the players arrive at the venue you can register them by creating a team entry (1 or more players per team) for a particular competition.

Matches overview: this tab shows a complete list of matches that need to be played. Grouped by competition/round and showing home team and away team.

Tables overview: tables available for use at the venue. Each table has a status so you can tell if it’s in use, or the players on it just happen to be practicing and causing delays!

Status: this page is where the benefit of using a software application comes into its own. At the top you can see the currently in progress matches. At the bottom you can see a list of matches that need to be played and if there are any free tables.

Omissions from this first demo?

Yes, there are some things missing and some things I’ll change as development progresses. For example, most tabs include a data entry area and an Add button which makes development/testing easier, but these will be moved into dialog boxes in time as they clutter up the main UI.

There’s also no obvious way to create matches between teams. This is coming, although it’s a bit more complex than other tabs as it needs to include a manual match creation and a randomised draw creation.

Results, I don’t see anywhere I can see match results! Again, an extra tab to be developed. The reason it’s not in the demo screenshots is that I don’t want it to be a tab in the same way as the other sections. I want it to be a separate window so those that run tournaments with a laptop and second screen can display the results window on the second screen. This allows players to come and see results (as well as the draw) without having to bug the organiser with questions.

Finally, the extra bits only possible by a software application. What if there was an internet forum you planned to update with results throughout the day? Wouldn’t it be nice if there was a Publish button that did everything for you – from logging into the forum, formatting the results and continually updating each time you click the Publish button? And, what about a website? A lot of organisers need their results published in a way they can put on their website.

The development continues …

Looping and reading a line at a time

while read line; do
  echo $line
done < test.txt

Remove annoying .DS_Store files even if directories that have spaces in them

find . -name .DS_Store -exec rm {} \;

Find differences between two directories (with subdirectories), stating if files differ

diff -rq $DIR1 $DIR2

Count files in current and all subdirectories

find . -type f | wc -l

I’ve not researched this a huge amount, so there is likely something out there that can provide most of my requirements. Initial searches show up web service solutions, but I need this to be local to the client machine or network for some channels of notification.

Requirements

• Subscribe to notifications from various channels with various levels of severity (thinking logging conifgurations)
• Fire notification events and let the framework handle distributing them to the subscribers

Implementation

• Configure new plugin wrapper via a configuration file – probably go down the Spring route for this
• OSX growl implementation using a native libgrowl library
• Windows growl implementation, libgrowl, using the java library that supports GNTP (Growl Network Transport Protocol)
• Java 6 system tray integration for taskbar notifications – looks crap on OSX

Progress

Initial concept developed and working, but not a reusable library

I came across a tool that does most of what you’d need, but it is based around US 9 ball tables. The tool is called Cue Table, although it is a bit buried within the forum section of the site.

Requirements

• Select a cue sport table to show the correct background (7×4 uk 8 ball, 6×3 uk 8 ball, etc)
• Drag and drop balls into position on the backgrond image
• Create line paths if ball movement needs to be displayed
• Create multiple scenes within a diagram if the scenario is better illustrated with multiple steps

Progress

Draft version created and functional, below is a rough idea of the save format.

<diagram name="Yesterday" width="800" height="450">
  <scene name="Default">
    <background image="7x4_uk8ball.jpeg">
      <coord x="0" y="0"/>
    </background>
    <icon name="red1" image="50mmRedBall.png">
      <coord x="100" y="150"/>
    </icon>
    <icon name="white1" image="50mmWhiteBall.png">
      <coord x="190" y="150"/>
    </icon>
    <path colour="red" arrowhead="end">
      <coord x="100" y="150"/>
      <coord x="790" y="155"/>
      <coord x="650" y="160"/>
    </path>
  </scene>
</diagram>