Scripting Snippets

Bits of scripting I use infrequently and always have to lookup!

Looping and reading a line at a time

while read line; do
  echo $line
done < test.txt

Remove annoying .DS_Store files even if directories that have spaces in them

find . -name .DS_Store -exec rm {} \;

Find differences between two directories (with subdirectories), stating if files differ

diff -rq $DIR1 $DIR2

Count files in current and all subdirectories

find . -type f | wc -l

Notification Wrapper

In a few of my projects I need to provide feedback using notifications, configurable by the user. What I need is a lightweight framework that allows a user to subscribe to notifications via various channels, e.g. growl, taskbar notification, email, sms, instant message, etc. Varying levels of severity, perhaps 5.

I’ve not researched this a huge amount, so there is likely something out there that can provide most of my requirements. Initial searches show up web service solutions, but I need this to be local to the client machine or network for some channels of notification.

Requirements

  • Subscribe to notifications from various channels with various levels of severity (thinking logging conifgurations)
  • Fire notification events and let the framework handle distributing them to the subscribers

Implementation

  • Configure new plugin wrapper via a configuration file – probably go down the Spring route for this
  • OSX growl implementation using a native libgrowl library
  • Windows growl implementation, libgrowl, using the java library that supports GNTP (Growl Network Transport Protocol)
  • Java 6 system tray integration for taskbar notifications – looks crap on OSX

Progress

Initial concept developed and working, but not a reusable library

Cue Sport Diagrams

If you’re creating diagrams of ball positions on a pool table it would be much easier if there was a dedicated tool rather than trying to use a paint package all the time. Whether you’re trying to show your friends a position from a recent frame or describing a practice routine, you’d need the same tools.

I came across a tool that does most of what you’d need, but it is based around US 9 ball tables. The tool is called Cue Table, although it is a bit buried within the forum section of the site.

Requirements

  • Select a cue sport table to show the correct background (7×4 uk 8 ball, 6×3 uk 8 ball, etc)
  • Drag and drop balls into position on the backgrond image
  • Create line paths if ball movement needs to be displayed
  • Create multiple scenes within a diagram if the scenario is better illustrated with multiple steps

Progress

Draft version created and functional, below is a rough idea of the save format.

<diagram name="Yesterday" width="800" height="450">
  <scene name="Default">
    <background image="7x4_uk8ball.jpeg">
      <coord x="0" y="0"/>
    </background>
    <icon name="red1" image="50mmRedBall.png">
      <coord x="100" y="150"/>
    </icon>
    <icon name="white1" image="50mmWhiteBall.png">
      <coord x="190" y="150"/>
    </icon>
    <path colour="red" arrowhead="end">
      <coord x="100" y="150"/>
      <coord x="790" y="155"/>
      <coord x="650" y="160"/>
    </path>
  </scene>
</diagram>

Backup Tool

Backup tools are not in short supply, but simplicity isn’t always a word I’d use to describe some solutions. I make extensive use of rsync, but without a simple equivalent on windows (although DeltaCopy looks promising) it’s hard to backup consistently across many platforms. Having the backup storage mirror the files so you can easily navigate and restore a lost file is also important.

This tool isn’t very high on my list these days due to suitable alternatives.

Requirements

  • Backup files to a local disk
  • Backup files to a remote disk over FTP
  • Backup files to a remote disk over SFTP
  • Backup files allowing individual file transformations, i.e. zip, encryption

Progress

Initial version developed, but poorly designed file wrapper class means rewrite is necessary

Dropbox Clone

Dropbox is a great idea with nice GUIs for multiple platforms, as well as a web interface. The problem is, if you’re a business you have to pay for a decent amount of storage – even 100GB isn’t enough sometimes. Plus, if your data is client confidential, the last thing you want to do is store it on a server out of your control. You could put it in a truecrypt vault, but backing up a binary blob isn’t great considering it will need an update for each slight change (this may be negated somewhat by Dropbox using rsync style technology for only transfering binary differences, not the whole file).

There is a project in it’s early stages, as I write this (Oct 2010), that can solve some of the problems – SparkleShare. It allows you to use your own server to store the files, removing the problems of insufficient storage space and using untrusted servers to store your confidential data.

That’s it right? Just wait for SparkleShare to reach a production level product? Well maybe … but I have more requirements for the environment I work in. We use a very large amount of disk space as we deal with video files, amongst other things. Employees tend to syncronise the parts of the remote storage they require to their local machine. Changes are re-synchronised on a ‘updated files only’ basis, but this is hardly full proof and only works at the moment due to the small number of employees with access. There is also no ability to backtrack changes, apart from resorting to the nightly backups. So, what would I like from a dropbox clone?

Requirements

  • Dropbox style folder on local machine that automatically stays in sync with the server
  • File versioning with the ability to backtrack changes (binary blob only, no subversion style change tracking)
  • Web interface for remote file access
  • File/folder filtering to allow the local directory to only synchronise part of the remote storage, while keeping the folder structure (remember it may be huge!)
  • Remote storage accessed over common file transfer protocols, such as FTP, or more likely SFTP

Implementation

  • Local file system change monitoring using JNotify
  • Hand crafted flat file DB system on remote storage to remove the need for a DB or server software. The clients will understand how to interpret the meta files stored alongside the actual data files/folders.

Progress

Planning – not started

JasperReports Tips

JasperReports is a java based open source reporting framework with similar, if not more, functionality to that of Crystal Reports. There is a visual report designer called iReport, but my experience of using it for anything but the simplest report has been a waste. It is useful for dragging and dropping report layouts around, but once you get the hang of it, it’s far easier to do it directly in XML.

Documentation is available, but it’s quite often difficult to get simple answers via a google search – as is possible for most open source software. The current books also lack good examples when they explain some of the more powerful features. Simple things like, I don’t want my report to have nulls printed on it, what are my options are suprisingly awkward to locate! My own, FAQs …

  • Null values in reports, what can I do?

    If you want to replace the null value with a blank entry this is easy to achieve with the isBlankWhenNull attribute of textField

    <textField isBlankWhenNull="true"> ... </textField>

    If you need to replace the null value with something else, you can do this directly in the report as shown below. Using the ternary operator the GROUP_NAME field is checked for a null, if it is null the text None is used instead.

    <textFieldExpression>(($F{GROUP_NAME} != null) ? $F{GROUP_NAME} : "None")</textFieldExpression>
  • How to add a row count?

    If you need a row count and don’t want to add unnecessary fields into the SQL statement you can use a variable. Straight after your field definitions include the following, assuming you have a field named USER_NAME in your fied definitions!

    <variable name="row_count" class="java.lang.Integer" calculation="Count">
        <variableExpression><![CDATA[$F{USER_NAME}]]></variableExpression>
        <initialValueExpression><![CDATA[new java.lang.Integer(0)]]></initialValueExpression>
    </variable>

    This can be used in the report as follow

    <textField>
        <reportElement x="0" y="0" width="30" height="20"/>
        <textFieldExpression class="java.lang.Integer"><![CDATA[$V{row_count}]]></textFieldExpression>
    </textField>
  • Highlight certain rows depending on the value of other data?

    If you need to highlight some text, the trick is to create two versions of the textField that displays it and use the <printWhenExpression> tag to determine which one displays the text. The below example highlight in red if the PASS_DATE field is null.

    <!-- main textField -->
    <textField>
        <reportElement x="466" y="0" width="228" height="20">
            <printWhenExpression>new Boolean($F{PASS_DATE} != null)</printWhenExpression>
        </reportElement>
        <textFieldExpression class="java.lang.String"><![CDATA[$F{USER_NAME}]]></textFieldExpression>
    </textField>
    
    <!-- highlight textField -->
    <textField>
        <reportElement x="466" y="0" width="228" height="20" forecolor="red">
            <printWhenExpression>new Boolean($F{PASS_DATE} == null)</printWhenExpression>
        </reportElement>
        <textFieldExpression class="java.lang.String"><![CDATA[$F{USER_NAME}]]></textFieldExpression>
    </textField>

JSTL Internationalisation

Some notes on how to get JSTL pages ready.

Setup

Inside a JSP page included on every other, for example a header jsp file.

<fmt:setLocale value="${param.locale}" scope="request" />
<fmt:setTimeZone value="${param.timeZone}" scope="request" />
<fmt:setBundle basename="Messages"/>

At your classpath root create a fallback localised file called Messages.properties in which to store key/value pairs. Files for other languages take the form of Messages_en.properties, Messages_en_US.properties, etc.

page.text.title=Page Title
page.text.message=Page Message
page.text.welcomeMessage=Welcome {0}
page.text.returnLink=<a href="{0}" title="Return">Return</a>

Examples

<h1>Page Title</h1>
<h1><fmt:message value="page.text.title"/></h1>

<h1>Welcome <c:out value="${person.name}"/></h1>
<h1><fmt:message value="page.text.welcomeMessage"><fmt:param value="${fn:escapeXml(person.name)"}/></fmt:message></h1>

Pull phpBB2 forum posts (basically a simple CMS)

phpBB2 is currently one of the most popular bulletin boards on the internet (yes, this statement is old, but the post is a repost of old content from my previous website).

Forum Powered Website

A forum is the most popular section of many websites so I was looking for a way to drive the website content from the forum. In the end I decided to write the following code to get topics from a forum which I could then display on other pages.

/*
 * PHPBB2 extension to get posts from a forum.  This allows you to power a news sections of
 * a website by posting in the forum.  Any posts in reply to the topic are classed as
 * comments.
 *
 * $Id$
 */

define('IN_PHPBB', true);

$phpbb_root_path = 'forum/';
$phpEx = 'php';

include($phpbb_root_path . 'config.' . $phpEx);
include($phpbb_root_path . 'includes/constants.' . $phpEx);
include($phpbb_root_path . 'includes/db.' . $phpEx);
include($phpbb_root_path . 'includes/template.' . $phpEx);
include($phpbb_root_path . 'includes/functions.' . $phpEx);
include($phpbb_root_path . 'includes/bbcode.' . $phpEx);

// need template defined as the bbcode uses it
$template = new Template($phpbb_root_path . 'templates/Saphic');

function &getForumPosts($forum_id) {
  global $db;

  $posts_array = array();

  // get list of topics
  $topic_sql = 'SELECT t.topic_id, t.topic_first_post_id, t.topic_replies FROM ' . TOPICS_TABLE . ' t WHERE t.forum_id = ' . $forum_id . ' ORDER BY t.topic_time DESC';
  
  if (!($topic_result = $db->sql_query($topic_sql))) { 
    return $posts_text;
  }
  
  if ($topics = $db->sql_fetchrowset($topic_result)) {
    $db->sql_freeresult($topic_result);
  }
  
  foreach($topics as $topic_row) {
    // get post for topic
    $post_sql = 'SELECT p.post_time, pt.post_subject, pt.post_text, pt.bbcode_uid FROM ' . POSTS_TABLE . ' p, ' . POSTS_TEXT_TABLE . ' pt WHERE p.post_id = ' . $topic_row['topic_first_post_id'] . ' AND p.post_id = pt.post_id';
    
    if (!($post_result = $db->sql_query($post_sql))) {
      continue;
    }
    
    if ($posts = $db->sql_fetchrowset($post_result)) { 
      $db->sql_freeresult($post_result); 
    }
    
    $post_row = $posts[0];  // should always have a result
    $text = bbencode_second_pass($post_row['post_text'], $post_row['bbcode_uid']);

    array_push($posts_array, array('topic_id' => $topic_row['topic_id'],
                                   'topic_replies' => $topic_row['topic_replies'],
                                   'post_time' => $post_row['post_time'],
                                   'post_subject' => $post_row['post_subject'],
                                   'post_text' => $text));
  }

  return $posts_array;
}

In the pages for your actual website, you can include the file above and then do the following.

<dl>
<?php
foreach(getForumPosts(13) as $post) {
  echo('<dt>' . date("jS F, Y", $post['post_time']) . '</dt>' . "n");
  echo('<dd>' . nl2br($post['post_text']) . '<br/>');
  echo($post['topic_replies'] . ' comment/s (<a href="forum/viewtopic.php?t=' . $post['topic_id'] . '">view</a>, <a href="http://www.example.com/forum/posting.php?mode=reply&t=' . $post['topic_id'] . '">add</a>)</dd>' . "n");
}?>
</dl>

Java Authentication – LDAP and Active Directory

Been asked to integrate your application’s authentication with an LDAP directory (Active Directory is LDAP v3 compliant)? Me too! There is a fair amount of information about this topic available by searching, but when I was doing this I couldn’t find one place that had everything explained in detail, so I decided to document how I did it.

What follows will explain how to validate a username/password combination against an LDAP compliant directory server using java and the opensource LDAP library called jldap.

First off, go and download the jldap jar file and browse around the code samples as it’s a well documented library. Second, take a look how easy it is to make a connection to an LDAP directory server.

LDAPConnection conn = new LDAPConnection();
conn.connect("localhost", 389);
conn.bind("cn=admin,dc=tilion,dc=org,dc=uk", "password");

If you’re not too sure about LDAP syntax you may like to read the Wiki LDAP entry. In short, LDAP uses a tree structure where each entry has a unique identifier, known as it’s Distinguished Name (DN). In the username above cn=admin is the Relative Distinguished Name (RDN) and dc=tilion,dc=org,dc=uk is the DN of it’s parent entry. Put together these form the DN for a user with privileges to bind to the LDAP directory (DC stands for Domain Component). Entries generally have a CN attribute, known as the Common Name along with a whole load of more familiar named attributes.

If binding to an Active Directory server, the username is more likely to be of the format cn=Administrator,cn=Users,dc=tilion,dc=org,dc=uk. Part of the complexity with LDAP queries is that there is no fixed format for where particular types of entry live from server to server. Most Active Directory servers will be alike, but won’t be the same when compared to a Novell directory or an OpenLDAP server. For the purposes of authentication we need to locate where in the directory the entries that represent a user object live.

  • cn=Users,dc=tilion,dc=org,dc=uk is the default for Active Directory
  • ou=People,dc=tilion,dc=org,dc=uk is the default an OpenLDAP server storing unix users accounts (the one I have anyway!)

Whoever set up the LDAP server should be able to tell you the base DN for your environment. For example, when setting up Active Directory you specify the name (check terminology) in the format machine.domain.ext, which would lead to the base DN of dc=machine,dc=domain,dc=ext.

Here is where we get to the querk. Imagine we need to check a login where the username is darren and the password, well, lets just say it’s the right password for the username. On an OpenLDAP server, all you need to do is try binding to the directory as shown below.

conn.bind("uid=darren,ou=People,dc=tilion,dc=org,dc=uk", "password");

uid is the attribute that holds the actual username value (the CN, or Common Name is often different to the actual username).

Unfortunately, Active Directory is different, in that you can only bind to it using a DN, which references the actual entry via it’s CN. What you have to do is perform a query to check if the username exists, grab it’s CN and then perform a bind using the CN and the given password. So, let’s see this in action …

// assume we have a connection, which is already bound
LDAPSearchResults searchResults = conn.search(
        "cn=Users,dc=tilion,dc=org,dc=uk",
        LDAPConnection.SCOPE_ONE,
        sAMAccountName + "=" + <username>,  // <username> came from the user trying to login
        null,
        false);
LDAPEntry entry = searchResults.next();
if (entry != null) {  // the username is valid, lets pull out the CN from the attributes
    String cnValue = null;
    LDAPAttributeSet attrSet = entry.getAttributeSet();
    Iterator allAttrs = attrSet.iterator();
    while (allAttrs.hasNext()) {
        LDAPAttribute attr = (LDAPAttribute)allAttrs.next();
        String attrName = attr.getName();
        if (attrName.equalsIgnoreCase("cn")) {  // we got the CN
            cnValue = attr.getStringValues().nextElement();
        } else {
            continue;
        }
    }

    if (cnValue == null) {
        // return auth failed, the username doesn't exist
    }

    // attempt a bind with CN and given password
    LDAPConnection tmp = new LDAPConnection();
    tmp.connect(HOST, PORT);
    tmp.bind("cn=" + cnValue + "," + "cn=Users,dc=tilion,dc=org,dc=uk", <password>);  // <password> came from the user trying to login

    // return auth successful, username and password are valid

    // an LDAPException is thrown if the credentials are invalid
}

Concepts covered, you’re probably wondering how are you going to find all those cn,dn,dc,xyz details about your particular LDAP directory? That’s exactly why I created a standalone application to query an LDAP server when I was learning this stuff. You can download the LDAP test application (NOT UPLOADED YET!), which includes the the compiled jar, full source and a maven pom.xml.

The code shown here is for illustration purposes only and should not be used in production without proper error handling additions. It is as concise as possible to illustrate a point.

Useful Attributes

A quick round up of useful attributes in various LDAP compliant servers.

Active Directory

  • sAMAccountName holds the username
  • displayName holds the full name
  • mail holds the email address

OpenLDAP (holding unix user accounts)

  • uid holds the username
  • cn holds the full name
  • mailacceptinggeneralid holds the email address

Computer networks and the internet for “normal people”

There are a huge amount of buzz words and confusing terms surrounding the internet and computer networking that makes it very difficult to understand for the everyday person. This article attempts to shed some light on the confusion and give you a broad overview of what happens when your computer connects to the internet.

Lets think about a network that most of us navigate and use almost every day without thinking it complex – the road network. The roads provide the way to get from source to destination and the cars provide the mode of transport. Keep that image in your head as you read through the following paragraphs.

Source and Destination

In the road network a source or destination can be thought of as a house or building address. In computing terms this is called an IP address and is of the form 123.123.123.123 – that is 4 sets of numbers from 0 to 255 separated by a dot. Other examples are 192.168.1.23, 10.0.0.1, 62.34.12.67 Just as each house address is unique, each IP address is unique.

How do you get an IP address then? This will differ from computer to computer depending on how yours is setup and what you use it for. One thing for sure is if you want to communicate on a computer network (like the internet) you will definately have an IP address.

Unlike a house address which never changes, your computer IP address may change from time to time. There is a limited set of unique IP addresses so they can’t just give them out to everybody. What normally happens is you lease an IP address for a set period of time while you need to use your computer and then free it up for someone else to use while you don’t need it. This process is hidden away from users so you don’t have to concern yourself with it. If you really want to know – something called a DHCP server is responsible for allocating you an IP address and telling you how long you can lease it for.

Domain Names

You’re probably thinking that you’ve never come across an IP address before yet you’ve been happily surfing the internet for months. Humans aren’t so good at remembering strings of numbers so we use domain names instead. Examples of domain names are www.google.com, www.bbc.co.uk and tilion.org.uk. Notice that they don’t necessarily have to start with www and they don’t all end the same (the endings are actually predefined and vary from country to country apart from the main three which are .com, .net and .org.

A domain name can be thought of as an easy way to remember an IP address. Servers known as Domain Name Servers (DNS for short) are responsible for converting the domain name you enter into a web browser, into the IP address to locate the machine hosting it. Again this happens behind the scenes and most users are totally unaware.

Transport

Unlike the road network, where you can see the cars travelling around from source to destination, a computer network sends electrical signals along wires (or wirelessly). The road network of the computer world is known as TCPIP and describes a way for your communication to travel around. All you really need to know is that if you try to contact another computer at a specified IP address (or domain name) the communication will succeed providing there are no broken roads between you and the destination.

Ports

Remember how an IP address identified a particular computer somewhere in the world. Most computers do more than one thing at once so there needs to be a way to differentiate between the services offered so the computer knows how to reply – this is where ports come in. An IP address can be likened to a house address, so a port can be likened to a way to enter the house (front door, rear door, window, sky light). A computer port is a number from 1 – 65535 (yep, it has a lot of ways to get in!).

Web Servers

One of the most popular services offered by other computers is serving web pages (known as HTTP or Hypertext Transfer Protocol). The HTTP bit is the way you have to speak to the server at the other end so it can understand you – think of it as speaking english or spanish and you must both be speaking the same one. Don’t worry too much about speaking HTTP as your web browser understands how to do that for you. Web servers typically listen on port 80 so when you type http://www.google.com into your web browser, 80 is the default port it uses to communicate with the other computer. http://www.google.com is exactly the same as typing in http://www.google.com:80 (the :80 mean use port 80). If you try to communicate on a different port the chances are noone will be listening or maybe the service that is listening doesn’t speak the same language as you, e.g. http://www.tilion.org.uk:22

Remember that the specified (or default) port is the one used on the server. Your computer is also using a port to talk to the server, but which one you shouldn’t worry about aside from the fact it will NOT be the same port as the server is using.

Other Services

Some other services you’ve probably used already and not realised are;

  • port 110 – POP3 which is used to get your emails
  • port 443 – HTTP in SSL mode which is a secure way to view a web page and is used for sensitive information like banking

Technical readers can probably spot mistakes in the analogies, but I’ve tried to keep technical detail to a minimum for the sake of understandability!