Thought-Crumbs

Subtle Subtitles clarifies conversation with people with dysarthric speech by presenting subtitles in our peripheral vision in an intuitively crafted wearable. Those familiar with the impediment understand the slurred speech however, as with other variances in expression, strangers struggle to ‘tune in’ to the conversation. Current solutions for this communication difficulty use synthesised speech to replace the person’s own; Subtle Subtitles ensures that the wearer’s own speech and eye contact continue to be of greatest importance, preserving the intimacy of the exchange whilst quietly providing a subtle hint of the context of the conversation.

Subtle Subtitles : Fashionable Technology for speech Dysarthria, Interaction Design Project from Calum Pringle on Vimeo.

From the offset I had the intention of designing a concept that represented my feelings of how the design of products for people with conditions such as Ataxia should be treated; enhancing rather than replacing reduced abilities; so the development of a working prototype was a happy bonus! 

My mum has Spino-Cerebellar Ataxia, diagnosed 7 years ago, and her voice is now affected by speech dysarthria, symptomatic of her Ataxia. Although friends and family still understand her for the most part, people who are less familiar are having some difficulty. This provided my project focus.

I started by researching the current state of the industry, looking for inspiration for the possible outcome of my project. After interviewing professionals in the field of inclusively designed products, and relevant therapists, attending charity and communication aid conferences, and following a workshop with the East of Scotland branch of Ataxia UK, I realised the potential of ‘subtitling’ speech with an iPhone / iPod touch application that was wearable. Having voice as the ‘input’ method as opposed to type or images such as is used in Toby Churchill’s Litewriter, means that instead of replacing a facility lost by the user, my system enhances the user’s remaining speech.

Inspired, I began to immerse myself in the group for which the project could be of use - people with Ataxia, with the accompanying speech impediment Ataxic Dysphonia / Dysarthria. Although my initial efforts with self-arranged charity meetings were attended by only a few, my role and interest in the charity Ataxia UK was a great way to get closer to other people with the condition. I was fortunately given the chance to attend a meeting with the East of Scotland Branch. I am indebted to the group for their help, as they became my user group for tests and feedback. I designed a series of exercises surrounding two packs of postcards. The packs contained typefaces and wearable products - some technological and others entirely ‘analogue’. 

(a booklet of results is available here (not exhaustive))

Through exploring the way the group perceived the fonts in conversational context (for example one exercise matched a typeface to a famous voice) I began to understand the route that I must take with my own typographic treatment to ensure that the scarf was not interpreted as either overtly authoritative or informal. 

The final typeface is in my own handwriting, styled to imitate the effect of chalk on black paper, used as a metaphor for writing that is not necessarily taken ‘word for word’. I have not, however, excluded the possibility of including other typefaces.

After having the opportunity to speak to Alan Newell, originally an engineer who developed one of the first LINK wearable communication devices, and the aforementioned research with other key stakeholders in the system, I realised that an iPhone application was a way of making the project not only flexible in how it could be displayed or worn, but more understandable in terms of the technology that we associate with it: which is ultimately a more universal approach to this design challenge.

With this grounding, I began to experiment with the (off the shelf) speech recognition software itself, and developed my first working prototype application in late December. 

The prototype employs Nuance Dragon’s speech recognition software on a windows laptop. Although the standard ‘training’ works well with ‘normal’ speech, it does not with moderately dysarthric speech. Working backwards, however, harnesses the power of the software by essentially removing some of the algorithms that are in place to help with writing (which is the original purpose of the software). This involves training the software each word individually, gradually building sentences. This is not practical for a commercial product, but demonstrates the potential of the software if it were to be re-worked. The microphone input ‘types’ into a text input box in a website. This site is checked continuously by javascript which, using ajax, sends any new words to a php script. These scripts then write the text to a file, which is then analysed by the javascript of a second website where the subtitle is finally displayed. Finally, using the jqtouch plugin for the javascript library ‘jquery’, the website is optimised for the ipod / iphone screen and packed into a prototype of a native application (basically a web app with a lot of javascript) using phonegap, which converts the site into an application and allows for further optimisation in xcode.

In parallel, I began role playing with my main user - this involved some hilarity with ipods taped to old scarves and some truly distracting typographic animations!

A realistation here, towards the end of the “zoom” phase over christmas, which has become of great importance to the project, was that for a system like this to work, we only need the occasional word to be subtitled. I since implemented further javascript that filters the words found in the text file to ensure that only the ‘key’ words in the sentence are subtitled. 

The user actually speaks too quickly for the system to be of any real benefit if every word was subtitled. I began to film the experience of instead subtitling only ‘key’ words by typing the input manually, and filming the results. This developed and became an almost weekly exercise, with the prototype maturing in tune to the feedback from each. This naturally progressed and the typographic animations became more refined. Anything other than a plain fade was confusing and meant that you were distracted from the person speaking. A side effect of this treatment was that longer words (above 13 characters) could not be displayed properly. Initially the solution was that of reintroducing some kind of movement to the display. It wasn’t until this was prototyped that I realised, as with conjunctions, words longer than 12 or so characters could happily be removed: it took the user long enough to say the word that the ‘listener’ would understand the word, without any subtitle aid. 

Once I understood the typographic treatment that was required, I began to refine the prototype application. I also made the decision to take my project down the route of crafting specially made wearables, as opposed to my alternative initial idea of making a ‘diy kit’ for modifying previously owned clothing. I felt this was best as I found that the correct location of the microphone and ipod was so important that it would be detrimental to the concept if they were implemented incorrectly.

I allowed myself some time to look into fashion design next, namely of scarves, ‘snoods’ and shawls, and sketched a couple of designs. It was at this stage that I essentially moved into the textiles department for two weeks, starting on the scarf itself. I decided to employ a chunky knit design as there was something unassuming and personal about the quality of knitted clothing, and wool is actually a very forgiving medium to work with. 

Once I had the knit, I began to ‘sew’ the scarf into shape to keep as much of the structure as possible. This has given the scarf a slightly more modern aesthetic and embraces the hidden microphone which also adds structure the shape of the scarf. 

To finish this phase I refined the ipod application and styled it to match the scarf; where I realised the opportunity to ‘skin’ the app in a way that matched the wearable. 

I also introduced the menus which show the intended ‘extendability’ of the application (to an extent). The final style of the application is ultimately decided by the wearer; it is up to them to photograph the wearable that they are to hold it in; not only does this make the experience more personal, but opens up the potential of having a range of wearables of different styles and materials. 

I ended this phase with an informal presentation of the project to the user group that had helped so much earlier in the year.

[presenting to users 15 march: http://blog.freecalum.co.uk/post/450802378/ataxia-uk-group-meeting-update]

Reflecting on the project, there have been several hurdles in terms of getting everything to work and even making the scarf, but everything has come together in a final piece that I could not be happier with. Although I am passionate about communication design, I am keen work anywhere in the field of interaction design as long as the work reflects my passion for inclusive, meaningful and thoughtful design, which I hope I have demonstrated in my execution of Subtle Subtitles.

My blog post introducing Subtle Subtitles is now featured on http://www.fashioningtech.com/ ! (or for the exact post - click here) but if you click soon, you’ll see it on the front page! 

Currently spending most of my time working on the degree show with postcard designs for the class:

Need to get these printed soon, and work out a design for holding them, kind of like a book style but a present…

I was looking out old pictures of the typeface I made for Subtle Subtitles, and realised that I should have decided on it far earlier than I did! In my initial pitch I used this image -

…And it took me months of research and user testing to come up with my handwritten, ‘chalk inspired’ font! I came to the chalk board metaphor as a way of communicating the written word but in a way that wouldn’t be taken as a definite thing. Basically embracing the potential of the system not always getting it perfect! 

Funny coincidence really. Annoying how inspiration was RIGHT in my face so early on! Ahh the creative process…!

Subtle Subtitles clarifies conversation with people with dysarthric speech by presenting subtitles in our peripheral vision in an intuitively crafted wearable. Those familiar with the impediment understand the slurred speech however, as with other variances in expression, strangers struggle to ‘tune in’ to the conversation. Current solutions for this communication difficulty use synthesised speech to replace the person’s own; Subtle Subtitles ensures that the wearer’s own speech and eye contact continue to be of greatest importance, preserving the intimacy of the exchange whilst quietly providing a subtle hint of the context of the conversation.