Dave's Bit Bucket

Thursday July 26, 2007

Automated Lip Reading

I watched a fascinating documentary on television the other night, which not only revealed some interesting information on an infamous historical figure, but also raised the curtain on a technology I'd expect to see much greater deployment of in the near future.

Please bear with me, and I'll describe the whole thing in context.

Between 1941 and 1944, Eva Braun - who had received some training in cine camera work and cinematography - shot a whole bunch of silent colour footage of the life that she and Adolf Hitler shared at the Berghof, the near-equally infamous guests they entertained, etc. This footage was discovered in 1945 by the OSS, who were looking for any evidence that would be useful to the prosecution in the Nuremburg trials. As the films were silent, they were considered to be of no evidential value and have remained, for decades, a simple historical curiosity.

A profoundly deaf computer scientist - who is, coincidentally, German - determined some years ago that computers could potentially use image analysis to lip-read, using the techniques that he employs day to day. He developed image analysis software which can, with a high degree of accuracy, map mouth, jaw and throat movements from captured video onto a computer-modelled head, and thus to phonemes. The software - which is currently optimised for German - is able to not only make such a mapping when the captured footage presents a subject face-on to the camera, but also when the subject has their face at anything up to 120 degrees from face-on.

Thus, with a little training (the details of which were unfortunately glossed-over, but which appeared to mostly involve identifying which area of the captured video represented a speaker's mouth), the software was able to translate the lip movements of the filmed subjects into speech components. Using the vocal talent of a number of German actors - one of whom was judged to impersonate Hitler particularly closely, based on the one covert audio recording of Hitler ever made, and the only recording of him in conversation, rather than giving a public speech - a vocal track was made to accompany the Berghof footage, and the two were dubbed together.

This was shown in the documentary, with English subtitles.

Now, consider the ramifications of automated lip reading (ALR) as applied in other contexts. If facial recognition software is able to not only identify a face but its components, such that it could pass details of mouth location to ALR software, if could become possible to reconstruct speech (and potentially and eventually, text) from high-resolution CCTV footage.

Given enough computing power, this could potentially be done for every face in a crowd.

Of course, caveats apply. German is a particularly clearly-enunciated language where every syllable is sounded, has a relatively small number of consistent pronounciation rules, and does not have variations in meaning associated with tonality. Tonal languages such as Thai, where a given small phonemal sequence enunciated in a tenor voice can mean something entirely different to the same phonemal sequence enunciated baritone, would most likely still require a skilled human interpreter to give meaning to the sound that ALR output would have, unless ALR is expected to eventually be able to interpret such things based on analysis of apparent constrictions in the footage of the speaker's throat.

Nonetheless, it's interesting...

Update: The technology on show was developed by Frank Hubner; I've also found a paper on automated head-recognition algorithms, designed specifically to facilitate mouth area identification for automated lip reading, here. Seems there's more folk working on this than initially meets the eye - it's a space to watch.

(2007-07-26 07:53:19.0) Permalink Comments [1]

"Integrity Checking in Depth"

Over the last few weeks, a number of folk have asked me about various elements of integrity checking and intrusion detection - so I figured it would be useful to aggregate my thoughts here.

First, I distinguish between integrity checking and intrusion detection, on the grounds that intrusion detection needs to be realtime whereas integrity checking does not; you can run BART or Tripwire a couple of times a day to gather reports of changes (this being the maximum recommended frequency, incidentally, as doing a full sweep of a system consumes a reasonable amount of system resource), whereas you'd want Prelude (or whatever IDS you're running) to scream at you in the form of an SNMP trap the moment it sees some activity it's not happy with.

With Solaris 10, you actually get the luxury of multiple methods of integrity checking; the title of this article is an allusion to the popular and valuable concept of "Defence in Depth", which is reflected in the ways in which you can make integrity checking work. The ways in which these integrity checking mechanisms relate to and complement eachother is subtle, and thus worth documenting. Out of the box, you get elfsign and BART, and with a little Internet connectivity, you can also call upon the services of the Solaris Fingerprint Database (SFpDB).

Today, elfsign verify <pathname> can tell you whether a given binary was genuinely released by Solaris Release Engineering as part of a production build of Solaris 10, or a subsequent Sun-issued patch to it. What it won't tell you, is if the binary is the current version, patch-wise; it also won't tell you anything about the config or change status of files which aren't ELF binaries, nor will it tell you about binaries bundled with Solaris which are part of specific third-party contributions (these don't get elfsigned by us, as from a legal perspective, signing a binary provided by a third party involves modifying that binary by changing its ELF header, thus creating a derivative work).

It is also worth noting that elfsign is a point of circular dependency. While anyone trying to trojan a Solaris 10 environment cannot get an elfsign-verifiable signature for their trojan binaries, if they were able - as a result of their attack mechanism - to trojan elfsign too, then it's Game Over. This is yet another good reason to deploy network-listening applications in sparse-root zones wherever possible, so that the elfsign binary remains usable but immutable from the application environment's perspective.

BART is a tool I typically describe - with tongue half in cheek - as "a poor man's Tripwire". While Glenn has produced a useful Blueprint on how to scale BART management, I still honestly think that, if you want to integrity-check a datacentre's-worth of installed systems, Tripwire is the superior product - this is owing to its cross-platform nature and the elegance of its key management. What both BART and Tripwire will tell you is, whether the file you had yesterday is the same as the file you have today. Both BART and Tripwire are agnostic regarding the file type, therefore they are applicable to scripts and config files as well as ELF binaries. However, they won't tell you where a file came from, nor what version it is. However, if someone manages to install a Trojan - or downgrade a Sun-issued and signed binary to a version which has an exploitable vulnerability which has subsequently been patched - either BART or Tripwire can potentially catch things which elfsign won't. The database of file digests which Tripwire holds on a system is also more resistant to attack than the equivalent BART database - the Tripwire db being signed with a key which is held on the Tripwire management infrastructure, rather than the host itself. Again, deploying network-accessible applications in sparse-root zones mitigates risk - most system binaries are loopback-mounted readonly in such environments, making them immutable even in the event of application compromise.

Last but not least, we have the Solaris Fingerprint Database. This contains comprehensive mappings between pathnames, MD5 digests and release / patch version numbers for scripts and binaries back to very early versions of Solaris; if you present a pathname and digest to it, it can tell you not only whether the associated file matches something produced by Solaris Release Engineering, but what release or patch version it is associated with. This maps to the functionality of elfsign plus the ability (when correlated with current patch levels) to spot binary version downgrade attacks, however Internet connectivity (direct or indirect) is required, and while it would be difficult to Trojan the MD5 digest generator to produce output which would match what the fingerprint database considered acceptable, it is not impossible (although sparse-root zones can help here, again).

So, there you have it - while any of the integrity verification tools above could potentially be compromised when used in isolation, owing to points of circular dependency (although to do so would be hard in all cases), a reinforcing combination of them plus the use of sparse-root zones results in an integrity enforcing and verification capability which would cause even a very capable attacker to have nightmares.

Glenn also has a Blueprint on integrating SFpDB and BART, which makes for good reading.

(2007-07-26 07:13:01.0) Permalink Comments [0]

Blindside

If you're not yet familiar with http://www.blindside.org.uk/, it's a blog / wiki site on which some interesting folk are recording some very forward thinking. The remit of Blindside is basically "take UK society in general, add technology (taking emerging technologies into consideration), and figure out what issues Government will face as a result, in 5 years' time or so".

Blindside is sponsored by the CSIA, and the Blindside folk are required to report their findings to CSIA on a regular basis. While the Blindside folk have a very large amount of experience, they acknowledge that they can't be experts in everything - so, particularly for the Wiki items, they encourage experts in the relevant fields to contribute to, and correct errors in, their records.

So, the bottom line is: if you care about UK Government's approach to technology, particularly technologies you are well-versed in, go to Blindside, read what's there, be intellectually stimulated and contribute your own wisdom, when you're not busy with your day job.

Thank you.

(2007-07-26 02:33:30.0) Permalink Comments [0]