Alan Burlison's Work Related Ramblings

Monday November 19, 2007

How to leave Facebook - followup 1

The electrons were barely dry on my last post when I received an email from TRUSTe about the problems I'd had getting Facebook to close my account; the interesting bit is below:

So my advice to you if you are having problems getting Facebook to close your account is to submit a complaint to TRUSTe.

How to leave Facebook

As I documented in my last post, it isn't actually possible to leave Facebook, all you can do is 'deactivate' your account. I got in touch with Facebook and asked them to delete my account, and here is the reply I got from them:

I wrote back to Facebook, saying that their response was unacceptable. I noted that their Privacy Policy page says that they are a licensee of the TRUSTe organisation, and that as such they are supposed to give users "choice and consent over how their information is used and shared". I also pointed out that as they are now registered in the UK, they are probably also subject to UK data protection legislation. Finally, I pointed out that Facebook had also been mentioned in a Channel 4 news report about identity theft, and that the media were obviously interested in Facebook's stance on data privacy and protection. I explained that if Facebook wasn't prepared to close my account I was prepared to take up the issue with the three avenues open to me, the TRUSTe complaints process, the UK Information Commissioner's Office (ICO) and the UK press.

In return I got exactly the same response as the one above. I wrote back to Facebook yet again, repeating that that their response was unacceptable, and that I was therefore going to take the three courses of action I outlined above. I registered complaints at both TRUSTe, the ICO and I also emailed Channel 4 News, explaining my story.

Last week Channel 4 came to interview me, and the item went out on Channel 4 News on Saturday 17th November. A video of the item can be found on the Channel 4 website. There's also details of the response from Facebook to C4's questions about their policy and process for account closures. Once the item had aired, I wrote again to Facebook, explaining that their response was still unacceptable, and that I'd taken the three options I'd identified in my earlier mail. Here's an excerpt from my mail to Facebook:

As well as sending my mail to the Facebook support person I had been dealing with, I also sent it to Chris Kelly, Facebook's Chief Privacy Officer, and Mark Zuckerberg, the Facebook CEO. Neither mail bounced, so I must have guessed their email addresses correctly. Earlier on today I received the following response from Facebook:

Hurrah! Although to be honest, this raises almost as many questions as it answers. If Facebook has the ability to delete accounts so easily, why don't they make it available to users? In their written response to C4 they say that "Facebook does not use any information from deactivated accounts for advertising purposes." If that is the case, why do they retain the information at all? And although they aren't using it for "advertising purposes", are they making other use of it, and if so, what?

I'm still waiting for responses from either TRUSTe or the ICO, I'll be sure to blog about them when I receive them. In the meantime, if you want to get Facebook to delete your account entirely, you can always try mailing them, quoting the clear precedent they have set by closing my account. I really can't understand why Facebook make the whole process so difficult, they are an extremely popular service and the amount of work involved in closing accounts properly is tiny in comparison to the volume of activity the site sees.

Friday November 02, 2007

Facebook and your lack of privacy

I've just attempted to delete my Facebook account, only to find this on the 'deactivate' page:

So quite clearly they DON'T actually delete your data, and I have been unable to find an option on the website to do this. I've emailed their privacy department, it will be interesting to see what response I get...

Monday October 15, 2007

Producing CSS with self-caching JSPs

One thing that's always bugged me about CSS is that it doesn't have any sort of macro facility, so if you want to use (say) a HTML colour code in several places you need to hard-code it into all of them, which is a right pain. As the app I'm working on is written using JSPs, the obvious thing to do was to output the stylesheet from a JSP, and then use EL to define variables and then insert them in the appropriate places, e.g.:

<c:set var="background" value="#292B4F"/>

body {
    background-color:   ${background};
}

That works just fine, but each time the stylesheet is referenced it is regenerated, and as the content is completely static that was obviously a bit inefficient. Some googling found a number of ways to cache the output of JSPs, from the simple to the extremely complex, e.g. using a servlet filter to intercept requests and cache output. I wanted the simplest possible mechanism that would do the job, with no external dependencies.

A fairly obvious technique is to cache the output in the servlet's application scope the first time it is generated, then on subsequent requests you just send back the cached output rather than regenerating the content. The basic outline looks like this:

<c:if test="${empty cachedStylesheet}">
<c:set var="cachedStylesheet" scope="application">

    <!%-- Cached content goes here -->

</c:set>
${cachedStylesheet}

That works fine, but if you watch the conversation between the browser and the servlet, the cached content is still refetched each time this is referenced. This is because JSP output is usually dynamic, so the servlet container sets things up so that the client browser doesn't cache the content obtained from JSPs. To fix this we need to manually add the appropriate headers to the HTTP response we send back to the client to that it knows to cache the content, and we also need to respond to requests from the browser asking if the content has changed. The relevant HTTP headers are If-Modified-Since, Last-Modified and Expires, see the HTTP specification for more details. This requires a little bit of additional inline Java code in the JSP, so the final version looks like this:

<%@page contentType="text/css; charset=UTF-8" pageEncoding="UTF-8" session="false"%>
<%@taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<c:if test="${empty cachedStylesheet}">
<c:set var="cachedStylesheet" scope="application">

<%-- Cached content goes here -->

</c:set>
<c:set var="cachedStylesheetDate"
  value="<%= new Long(new java.util.Date().getTime()) %>"
  scope="application"/>
</c:if>
<%
  long date = (Long) application.getAttribute("cachedStylesheetDate");
  long mod = request.getDateHeader("If-Modified-Since");
  if (mod > date) {
      response.setStatus(HttpServletResponse.SC_NOT_MODIFIED);
      return;
  }
  response.setDateHeader("Last-Modified", date);
  response.setDateHeader("Expires", new java.util.Date().getTime() + 86400000);
%>
${cachedStylesheet}

Here's how it works: We cache the content the first time the JSP is accessed as before, but we also set a session scope variable cachedStylesheetDate that records the time when the output was generated. Then on each request we fetch the value of any If-Modified-Since header specified by the browser. If the content was generated before the date, the browser already has an up-to-date version of the content, so we just send back a 304 response (Not Modified) to indicate that fact. Otherwise we set the Last-Modified header to the date when the content was generated, and then set the Expires header to tell the browser to check again in (60 * 60 * 24) * 1000 = 86400000 milliseconds, i.e. in 24 hours, and then we send the cached output. That way, after the initial fetch, the browser will only check once a day to see if the content has been updated - that figure can of course be adjusted as necessary.

Saturday September 29, 2007

The new opensolaris.org user management webapp moves into the light

As outlined here I've been working away on providing the infrastructure needed to break up the current monolithic J2EE application used to host opensolaris.org. I've just released the source code of the existing portal, a large J2EE application. As detailed in the restructuring document, the intention is to split the functionality of the current webapp into smaller, more manageable parts. As I was releasing the portal code, I thought it was a good time to move my development repository outside as well. Accordingly, you can browse the source at:

http://src.opensolaris.org/source/xref/website/auth/trunk/

and the source can be accessed via anonymous SVN at:

svn+ssh://anon-AT-opensolaris.org/svn/website/auth

It is *very* rudimentary and is *far* from being finished. What I've put on opensolaris.org is just a snapshot of my development workspace as of today. However the database schema is available, along with a small set of dummy data which is used to populate the Derby database currently used by the prototype webapp.If you have any questions, I encourage you to subscribe to the website-discuss mailing list and ask them there.

Posted by alanbur ( Sep 29 2007, 12:48:25 AM BST ) Permalink Comments [3]

Source code for the opensolaris.org portal released

After a long haul, I've finally released the source code of the J2EE web application that runs the opensolaris.org website. If you are interested, the source can be browsed online at:

http://src.opensolaris.org/source/xref/website/portal/trunk/

and the source can be accessed via anonymous SVN at:

svn+ssh://anon-AT-opensolaris.org/svn/website/portal

If you have any questions, I encourage you to subscribe to the website-discuss mailing list and ask them there.

Enjoy!

Thursday August 23, 2007

Acrobat on Solaris x86 - a hopeful sign!

The folks over at Adobe have just set up a blog entitled Adobe Reader on Unix, and the first post says:

Which is a very hopeful sign - the version of Acrobat which is currently available on Solars x86 is ancient, and I'm certain an up-to-date version would be warmly welcomed all round. Why not head over to the Adbobe blog and give them some encouragement? ;-)

Thursday June 21, 2007

Greg P on the BBC

Just noticed an interview with Greg Papadopoulos, Sun's CTO in the Technology section of the BBC News website.  The interview is about Greg's views of future technology trends.  He makes some interesting points about mobile phones, PCs, the inexorable rise of the network, and kitchen utensils :-)

Thursday June 14, 2007

Visiting Baby - the world's first stored-program computer

I had to attend a meeting in Manchester on Tuesday afternoon, so while I was there I took the chance to drop in on Baby, or as it is more properly known, the Small-Scale Experimental Machine, or SSEM for short. Baby was the world's first stored-program electronic digital computer, running its first program on June 21 1948. Now before all you US citizens start getting all uppity and start telling me that ENIAC first ran two years earlier in 1946, the unique and ground-breaking feature of Baby is that the program which it ran was stored electronically and in the same store as the data, making it the first machine with a von Neumann architecture - Baby is therefore the ancestor of every modern computer. ENIAC used decimal arithmetic and was hard-wired - changing the program required a lady with a pair of pliers. And anyway, ENIAC was beaten by the British Colossus (1943), which was both binary and electronic, but not Turing-complete, and in turn Colossus was beaten by the German Z3 machine (1941) which was Turing complete and used binary arithmetic, but was electromechanical.

Right, now we've got the historical arguments sorted, on to the machine itself. The machine was constructed at the University of Manchester by Frederic Williams, Tom Kilburn and Geoff Tootill. During WWII, Williams and Kilburn had both worked at the Telecommunications Research Establishment, which was a cover organisation set up to do work on radar.

The ground-breaking feature of Baby was the Williams-Kilburn tube. This was a cathode ray tube which formed the memory of the machine. Williams and Kilburn had extensive experience of CRTs from their experience of developing WWII radar equipment, and in fact the tubes they used in Baby were from radar equipment. The glass surface of the tube was used as the store - the electron beam was used to write a grid of charged spots on the glass, with different charge levels representing 1 and 0. The values were read off via a metal plate on the outside of the tube. The charges leaked away over time, so the contents had to be refreshed on a regular basis, in a very similar way to modern DRAM. Baby used a 32 x 32 pattern of dots, giving a memory capacity of 32 words each of 32 bits. Williams-Kilburn tubes provided random access to the stored data, unlike the alternative technology of the time, the mercury delay line, which only provided serial access. Baby actually used four tubes, one as the main store (memory), one as the Accumulator, another to hold the address of the current Instruction (CI - Control Instruction) and the instruction itself (PI - Present Instruction). The final tube was used to mirror the contents of the other three, the tube displayed being switchable. The display tube was necessary because the other three tubes had a pick-up plate on the front, and they were also heavily shielded to prevent electrical interference.

The instruction set used bits 0-12 of a 32-bit word to hold the target address, and bits 13-15 to define the operation. The instruction set was very simple with just 7 instructions:

• Jump indirect
• Relative jump indirect
• Take a number from memory, negate it, and load it into the accumulator
• Write the number in the accumulator back to memory
• Subtract a value from the accumulator
• Skip next instruction if accumulator is negative
• Halt

Note there is no addition operation - you can implement addition using subtraction and negation, but you can't do the inverse. The reason for the limited memory and instruction set is because Baby was intended purely as a technology test-bed. The work carried out fed directly into later machines such as the Manchester Mark 1 and the Ferranti Mark 1. In 1995 the decision was taken to build a complete replica of Baby in time for the 50th anniversary celebrations in 1998 and this effort was successful, with the resulting replica being housed in the Museum of Science and Industry in Manchester. The machine is powered up every Tuesday, hence my visit.

The picture on the left shows the entire machine. The rack on the far left contains the power supplies, the rack on the far right contains the storage CRTs. In the centre is the panel containing the control, switches and display. The remaining racks contain the machine's logic circuits. The photo on the right shows a close-up view of the control rack. At the top is the display CRT - if you look carefully you can see that a program is running, an animation of a ship is visible at the bottom of the tube, sailing from left to right. The knobs to the right of the tube allow you to select the tube which is being displayed. Immediately below is the 'typewriter' which is used to input the program, each switch corresponds to one bit. There are 40 switches, so 8 of them are unused. Below that is a panel containing switches to select the line number in the store to modify (top) and switches to select the Function (opcode) to be executed (bottom). Finally, at the bottom of the picture is a row of control switches used to clear, load, run and stop the program. Click on either picture for a bigger view.

Great pains have been taken with historical accuracy - the people who worked on the original machine were consulted wherever possible, and many hours were spent poring over original notebooks and old photos, identifying the exact placement of component and labelling. The same components as were used in the original machine were used throughout. Even the racks are authentic, even though some of them had to be rescued from someone's garden! The racks are standard Post Office ones with 19 inch mountings - anyone recognise that dimension? ;-) They even went as far as replicating the numbering on the bottom of the racks, although nobody knew what the numbers were, and the chances were that the racks were second-hand in the original machine. My favourite touch is that in the original machine the display CRT (top of the right-hand picture) was propped up on a cardboard valve box as the cutout in the panel was too big - that's been replicated too!

The first program to run on the machine was written by Tom Kilburn, and calculated the highest factor of 2¹⁸. One of the other early programs written for the machine was a long division routine written by Alan Turing who was working at the National Physical Laboratory at the time, but who was shortly to move to Manchester University.

I asked how reliable the replica was, and remarkably it seems to be very reliable, unlike the original version. It seems that the main reason is that the valves in the replica are mostly 1950s and 1960s vintage, and manufacturing techniques improved rapidly in the period after the war. The machine is run every week, and in the last 10 years only 3-4 valves have failed. More problematic are the old wire-wound resistors, which tend to fail more often.

To coincide with the 50th anniversary there was a programming competition. As a result several simulators are available, along with a programmer's reference manual and example programs. My favourite emulator is David Sharp's, which has a graphical interface that is similar to the real machine. It is written in Java, and I've been working with David to produce a Webstart version (not available yet) as well as trying to improve the visual accuracy of the simulator.

It was fascinating to read some of the documents and papers that have been written about the machine, and to see the machine actually running. Many of the concepts and techniques we still use today such as dynamically refreshed memory and relative jumps first appeared in this machine and it is amazing when you realise just how architecturally similar the SSEM is to modern machines, despite its appearance. I knew that Manchester had played a part in the early history of computing, but until I spent some time reading about the SSEM I hadn't realised just how pivotal that contribution had been.

Tuesday May 29, 2007

Someone had to do it

/dev/bollocks goes Web 2.0

The birth of a new programming language

A rare chance to watch (and even contribute to!) the birth of a new programming language - lolcode.com. 1337!

Thursday May 03, 2007

XML-based J2EE frameworks considered harmful

Are you using one of the XML-based web frameworks such as Spring, WebWork or Struts?  Then you've been duped.  Conned. Flimflammed. Bamboozled. Hornswoggled.  (Yes, this is a rant ;-) Here's why:

• XML doesn't get executed, it gets read
  Unlike Java source code, XML just gets read once rather than being executed.  That means that it can't respond dynamically to changes in the application environment. You are stuck with whatever is configured at the time the XML is read - usually this is at the very beginning of your applications startup processing.
  
  
• XML doesn't have constants or variables
  Want to use the same configuration value across multiple XML files?  Tough, you can't.  The only way you can get even remotely near to this is to use vile preprocessing hacks.
  
  
• XML doesn't have flow-of-control
  Want to put some conditional logic or loops in your XML?  Sorry, there's no standard way of doing this.  Sure, some frameworks attempt to provide this using their own custom XML entities, but every one of them does it differently.
  
  
• XML files aren't programs
  All that good stuff you were taught in your CS classes about type safety, information hiding, coupling, cohesion and so forth?  Forget it, thanks to XML you are now back in the 1960's.
   
• Welcome to action at a distance
  See that attribute you set in your XML?  Well, that probably has to map onto a method call on an object.  Which method and which object?  Prepare to spend hours pouring over the source of your framework to find out.
  
  
• You can't garbage-collect XML
  See all that XML goop you've lovingly crafted?  Well, say goodbye to large chunks of the memory on your machine.  The application frameworks usually like instantiate all of the objects you might ever need at startup, because they have no way of knowing if they are ever used.  And the frameworks make sure they are nailed into place, so forget about them ever being GC'd if they aren't actually used.
  
  
• You'd better understand all about classloader leaks
  All this XML you've used requires all sorts of introspective sleight of hand by your application framework.  And that involves getting it's grubby little hands into the Java runtime support mechanisms, and holding references to bits of its innards such as classloaders.  And the next time you redeploy your application, all those references probably aren't going to be cleaned up properly, so say goodbye to lots of leaked memory.  And if you redeploy just a few times more, expect your application to fall over completely as it runs out of memory.
  
  
• You've defeated type checking
  Most everything in XML ends up being a text string.  That includes class names, method names and method parameters.  Got one of them wrong?  Oops, more mysterious exceptions will be heading your way at some random point in the future.
  
  
• Debuggers don't debug XML
  That cool debugger your IDE has?  Don't bother.  It doesn't work on XML files.  The important bits of your application logic are now buried in some introspective labyrinth buried deep inside a library that you probably don't understand fully.  Welcome instead to the world of cryptic stack traces and sleepless nights.
  
  
• You can't run FindBugs or PMD on XML
  Those nice tools that check your source code for programming errors?  Guess what?  They don't work on XML.  You've just thrown away another tool in your armoury.
  
  
• You've moved bug detection to run-time
  All that wonderful checking that the Java compiler does to make sure your application is correct?  Gone, javac doesn't do XML.  You aren't going to know if your application is even minimally correct until you try running  it - and there's a fair chance that it will run for some considerable time before you find out that you've made some simple programming error or other.
  
  
• You still have to rebuild your application if you modify it
  XML proponents tell us that one of the major benefits of XML for configuring J2EE frameworks is that you can reconfigure your application without changing the source.  Really?  How? Does anyone ever change the XML files in a deployed WAR file?  Of course not, that would be insane.  Ah, you have to edit the source XML files then rebuild and redeploy your WAR file - Oh dear, you still would have had to do that even if you didn't use XML at all.
  
  
• You've just doubled your troubles
  Think that Java is a big, complicated language?  You 'aint seen nothing yet. Faced with a big complex application and want to figure out who is calling a method?  Well, don't expect your IDE to be of any help, it doesn't understand your application framework's XML files.  Trying to trace through the code and figure out what happens?  Forget it.  You'll go insane hopping between Java and XML source and trying to figure out which bit of opaque magic provided by your application framework is doing what, where and when.
  

I'm fully aware that the list above is crammed with generalisations, and that there are various hacks and workarounds for some of the issues.  However the overall point I'm making is that XML isn't a programming language, yet virtually all of the major J2EE frameworks use it as if it is.  The pervasive use of XML for tasks for which it is not suited has effectively discarded the last 30 years of software engineering advances. This is a huge mistake, and I expect that in 5 years time people will look back at the current XML mania and say "What the hell were we thinking of?"

Thursday April 05, 2007

Why I hate XML configuration files

I'm working on a reasonably large (1000+ source files) J2EE application that makes extensive (some might say utterly excessive) use of external components - it requires in the order of 50 JAR libraries to run, over and above the standard J2EE ones.

And of course many of those JAR files have their own unique XML files to configure them.

And of course many bits information related to the configuration of the application (URLs, database details etc) have to be repeated in more than one of those XML files.

And there's no global way of doing this.

So the people who originally wrote the application came up with a scheme - they'd store the configuration values in property files.

But that only helped for things that got the values dynamically, for example by using the J2SE Properties class.  All those external components they'd used didn't know anything about the application's property files, they only knew about their own unique little XML files.

So the people writing the application came up with another scheme - they'd embed tokens in all those little XML files, then use the Ant Filter task to replace them at build-time with the values from the property files.

"Job's a good 'un!" they doubtless exclaimed, flushed with their extreme cleverness.

Then I came along and had to maintain the beast, when I found:

• There are in excess of 50 of these properties, which makes configuring the application more than a little tedious and more than a little error-prone.
• The names of the tokens processed by the Ant Filter task and the names of the properties used to set their values mostly bear very little relationship to each other.
• There is no consistently-applied property naming scheme.  There are in fact several inconsistently-applied naming schemes.
  
• Some of the properties are just renamed versions of other properties.  Sometimes they are inconsistently renamed versions of other properties. Sometimes they don't appear to be used at all.
  
• It isn't clear when reading a property file which properties are only used at compile time by the Ant Filter task, and which properties are only used at run time.  It's also unclear  which ones are used at both compile and run time.
  
• The compile-time use of properties is a killer because it means the application can't realistically be reconfigured without recompiling it - because the use of the Ant Filter task means the compile-time property file values end up hard-coded in many different places.  It's therefore pretty much impossible to produce a version of the application that can be deployed on more than one machine - for example for development or test purposes.
  

All this means that there's a whole series of bear traps set for anyone who innocently changes any of the properties files of the application, in the mistaken assumption that the application will actually take any notice of them.

I'm sure I'm not the only person to have been hit by this problem - how to configure multiple external components which all have their own XML configuration files - without having to hand-edit each of the XML files each time every time something changes, and without hard-coding the configuration at build time.  However I'm damned if I know of a good way of doing it - although I can easily think up several not-very-good ways of doing it.

If you know, please let me know!

Tuesday January 16, 2007

It's official - Sun is the #1 contributor to Open Source - by a long way

Just seen this link posted internally: the rather windily-named Study on the Economic impact of open source software on innovation and the competitiveness of the Information and Communication Technologies (ICT) sector in the EU. To cut to the chase, Sun is acknowledged as being the number one contributor to Open Source, outstripping the second contributor (IBM) by nearly 3½ times.  I've reproduced the relevant table below:

Table 5: Cost estimate for FLOSS code contributed by firms

And that's before the recent OpenJDK announcement!

Friday December 01, 2006

A useful little web access log reporting tool

One of the things that was on my to-do list after setting up the Meninos do Morumbi Oldham website was to do something on the reporting front with the server log files.  I'd already set up Tomcat to generate combined log format files by putting this in the server.xml file:

    <Valve className="org.apache.catalina.valves.AccessLogValve"
      directory="logs"
      prefix="access."
      suffix=".log"
      pattern="combined"
      resolveHosts="false" />

so I had the raw data I needed, I just needed to do something with it.  In the past I've used AWStats to do log file reporting, but it is written in perl and therefore needs a CGI-bin setup.  This is easily done if you are running Apache, but I'm running stand-alone Tomcat, and although you can run CGI stuff under Tomcat, it isn't really recommended.

As is often the way, I was looking for something else entirely when I came across Visitors, a stand-alone log file analyser written in C.  It writes its report as either a single HTML or text file, and the report looked fine, so I could just run it from cron once an hour and put the generated report somewhere in the tree managed by MeshCMS.

One small additional wrinkle: as you can see from the sever.xml entry above I've turned off DNS lookups for the access logs.  The reason for this is that DNS lookups can take some time, and I don't want logging to slow down the web server.  However it's useful to have resolved names for reporting purposes, so I pipe the log files through the apache logresolve utility before feeding them into Visitors.  At the moment I'm doing this each time I build the reports - I should really just do this once and cache the result, but that's a job for another day :-)