When one coach is not enough…

The main idea of the EU Horizon 2020 project Council of Coaches is to have a number of coaches that you can gather and meet. I think it is a clever idea (you never know where playing video games are good for…). Why?

Here I present the challenges I see in this project regarding reseach in the use of virtual conversational characters for serious applications (other than demonstrators, gaming or art) . There is an extensive project website containing a lot of information.

Goal of the project is to help elderly people like me (for practical purposes: 55+ and you are old nowadays; and young as well) to reflect on their health issues. Specific target are the chronic diseases: lung-diseases (astma; copd) , heart-diseases, chronic back pain; obesitas, diabetes. If you happen to suffer from one of these the others will most of the time come soon. How to cope with this situation?

To be honest – and why shouldn’t I ? – I do not know how many people (in the target group) reflect on their health issues so that they search for help. And if they do what is the trigger and what are their needs and where do they go for help. I myself, I call the doctor if I think there is something not good with my body; maybe after I searched on the internet to see what information I can find. We had a chat once with an internet doctor when we were abroad and needed some medical advice. When we have a lasting back pain we go to a doctor. The doctor says: it’s the age. Ageing is slowly dying. You have to live with that. Or die, if you want. We are getting older and I think we are not special.

Digital coaching app all over the place

There are digital personal diabetes coaches in the form of an app that runs on your mobile phone. Some of them have an embodied character that pops up when you want. They try to motivate you to measure your glucose regularly. There are physical activity coach apps. There are food coaches; sleep coaches, depression coaches; budget coaches. The most important function is that they help you to monitor your physical or financial condition in terms of some vital physiological or financial parameters and some of your daily health-related activities. And sometimes they can give you personal advice. Your values are either obtained from sensors connected to the app (step counter, glucose meter), from information you provide yourself by typing in values in form fields, from other applications you use. And sometimes from a short information dialog with your personal coach. But this is still a real challenge: to have a free, open, natural interaction in your own language with an artificial conversational agent that is really of your help.

I am rather sceptic about the potentials of human-virtual human-interaction. The value of apps is that they store relevant data, they can help you reflect on a specific health related issue: your weight, your diabetes. People that use such an app for a longer time are already motivated to keep an eye on their health condition. You data can easily be shared with your doctor. Older people forget things; it helps you remember. Ease of use and functionality is what counts. Should it be fun to work with an app? I think it is a nice added value. But it should not undermine the primary functions. For fun there are games.

Serious coaching games

Children like games. In one of the Human Media Interaction projects at Twente University we built a gamification platform to support young diabetes patients in dealing with their disease. In a journal paper we discuss what barriers we encounter on the path from design to the final implementation and inclusion in the current health ecostructure. Some elderly people like games as well. So why not design serious games that help people with their personal issues in a challenging way?

The very idea of the Council of Coaches project comes from the world of video games. Different coaches, covering expertice in various domains of life, chat about some issues relevant for and brought in by the user. This way the user need not be actively contribute to the conversation all the time. She can jump in whenever she likes.

One of the big challenges of the project is to get content for the health dialogs. How to feed the virtual coaches so that they are able to contribute in a sensible way to a conversation about the personal issues raised by a user? Maybe interaction and cooperation with real public coaching sessions can be of help.

Health Insight:  Council of Coaches as Interactive TV format

On the list of Most Important Things For a Good Life a good health condition seems to be number one. But football is definitely second. The most popular and most awarded TV production in the Netherlands for many years already was Voetbal Inside. A council of four football (soccer) coaches discusses the most important issues of the week. TV watchers can drop a line via social media and ask questions, often addressed to one of the council members. The issue selected by the moderator is shown on screen and lively discussed by the different characters of the council (see Figure X). It is a mixture of spontaneous live and scripted interactive TV. Sometimes direct video communication with a guest/watcher is broadcasted.

Why not exploit Council of Coaches as an Interactive TV format. The Dutch Omroep MAX  would be the first to be interested. They target the older segment of the Dutch TV watchers. They have close contacts with care institutes for elderly people. Most popular program of Omroep MAX is the cooking/bakery contest “Heel Holland Bakt’’  in which “normal’’ people take part.  Omroep MAX might be interested to cooperate in a TV series where normal people can discuss their health related problems (focusing on a specific disease or general health related issue: diabetes or obesitas) with the Council of Coaches on TV. The Council consists of well-known medical experts and other “Well known Dutch’’ that suffer from the disease of the week. The recordings together with the feedback about engagement (e.g., audience ratings) can be used as training material for home-made artificial coaches. TV doctors we have already for quite some time but a council of coaches that discusses a statement like “Doctor, I have diabetes could it be because of stress in my work?’’  (see for an answer: https://www.dokterdokter.nl/gezondheid/tv-dokter/page/2/    )   could be a valuable addition. For the society it would be a welcome contra-weight against all those media productions and commercials that promote the food industry.

Figure 1:  The Council of Football Coaches, one of the most popular interactive TV productions in the Netherlands about the second most important thing in men’s life.

Figure 2. Insight. The Council of Coaches on TV. 

Figure 2 shows the Council of Coaches on TV. The picture on top shows a scene from the popular Dutch OmroepMAX TV production “Hendrik Groen’’ about a group of elderly people in an elderly house. The character on the right is Hendrik. The character on the left is his best friend Evert, a diabetic. Evert likes to drink, maybe a bit too much. He just had his leg cut off in hospital because of gangrene (necrosis caused by diabetes).  At the bottom you see the question under discussion. The council members are known Dutch TV personalities and experts in food, diabetes care.  They discuss the problem and conclude that it would be good that diabetics find out for themselves how their blood glucose values are affected by the alcohol consumption because. TV watchers can download the COACH app, that can be connected to their glucose meter.  The app allows them to chat with their personal diabetes coach, a virtual replica of one of the council members (they lent their voice and style to the character).  The coach gives them instructions how to perform the test and motivates them to adhere to the protocol for the duration of the test and to keep track of their alcohol consumption. Outcomes are sent to the COUCH server and shared with the audience in a following episode of Health Insight.

Spoken dialog with artificial characters: a real challenge

The problem of real-time automatic speech recognition is, and will remain, “close to being solved’’, thanks to Big Data and DNN technology. Real-time is a necessary requirement for spoken dialog that doesn’t suffer from processing delay and that allows realistic turn-taking and interrupting behaviour.  One big problem is the recognition of special utterances, named entities, and newspeak.  Data used for training machines is typically historical and outdated. Hence the need for continuous updates. As Hugo Brandt-Corstius – one of the founding fathers of Dutch research in formalisation of natural languages – used to say, “Wat je ook doet, de semantiek gooit roet.” (“Whatever you do, semantics bothers you”).

The generation of natural live-like speech is ready to be exploited by virtual characters (see: https://deepmind.com/blog/wavenet-generative-model-raw-audio/ ) so it is possible to have a personal coach with the voice of, for example, André van Duin (Evert in the Hendrik Groen TV series) or Trump to name a trustworthy figure.

An unsolvable paradox

But the core problem of an artificial dialog is the logic of the conversation. There is none. And if there is some logic it is the participants themselves who decide what it is. Not the designer. Trying to design a system for natural open dialog is trying to solve a paradox. Being a conversational partner you do not want to control the other party’s response. Of course when you ask a question you more or less force the other to respond in some way or another. But not in a deterministic way. That’s the whole idea of a question, isn’t it? Getting to know someone is different from asking all kinds of information about or from someone. The best realisable technical system offers the users a number of options. It also has a number of options the system can choose from to respond to a user’s action. These systems assume by desing that the world of conversations is closed; that there is something like a mathematical space of all possible conversations. That language is a system. I believe it is not. Autonomous agents ignore the users’ freedom, their identity and autonomy. The very concept of user already challenges these human values.


Modern projects deliver demonstrators. Project reviewers don’t like to read reports or scientific publications. Show me.

The Council of Coaches project built a functional demonstrator. The world of possible conversations is designed using a dialog editing system, called WOOL also developed in the project. So every conversation that a user and the coaches in the council have is a realisation of one of a huge set of possible paths of pre-scripted dialog continuations.

In another technical demonstration system embodied virtual 3D characters simulate realistic multi-party conversations. It demonstrates progress made and state-of-the-art in the development of turn-taking, addressing and argumentative dialog behavior generation for artificial conversational embodied 3D characters.

Societal Challenges

In his “Computer Power and Human Reason’’ Joseph Weizenbaum tries to understand how it is possible that people interact with his computer program ELIZA as if they are talking to a real human. The psychological value of interaction with artificial characters is also the theme of Sherry Turkle’s “Alone together”. As long as people recognize themselves in the answers given and as long as there is sufficient room for interpretation for the user her experience of being recognized by the system is strengthened.

Council of Coaches is a project that builds bridges between new media, art and design, and technology.

How will people stand towards virtual health coaches? Will they see them as personal coaches that they are willing to share their personal experiences and part of their life with?  Do members of the council have to say only what we (or the medical people) believe is correct or do we allow bad or sceptic characters?  Is the system seen as a medical instrument? Or is it a system that tries to make users aware of the way they stand towards their own life in whatever way that we believe that works?

Moreover, do users want to have private discussions with one of the virtual coaches? May a coach deceive the patient or withhold information because he believes it is not good for the client’s health to know? Old issues in medical ethics get a new dimension when coaching in the health domain becomes virtual. What is new is that many users are inclined to uncritically believe what the computer says: “The computer told me!”. These are some of the societal issues that have to be considered before the Council of Coaches will find its place in the social organization of patient-centered health care.

Future dream and worst scenarios

Some people prefer to talk about their personal health issues with a virtual character instead of talking to a human expert. Others are more or less forced by the health care system to first chat with an e-health coach before they see a real human. I am not sure if this is a healthy thing. Maybe the Council of Coaches can help the users to identify their personal issues and brake the barriers to talk to real humans.

A worst case scenario would be, when society decided to replace human coaches and experts by artificial agents because of the economic burden of a human good quality health care system for elderly people. Many people have the impression that western society is moving in the direction of this worsest scenario rules by policies that have unlimited trust in autonomous artificial intelligent agencies.

The Council of Coaches project has delivered a proof of concept and a software platform and tools that can be applied for building end user applications in other domains, for example for social skill training in professional organisations.

The real danger of autonomous agents

Regarding the discussion about “autonomous technology’’ (social robots, killer robots, autonomous cars, virtual coaches that would take the place of real humans): some people see a danger in the growing number and autonomy of intelligent machines that would take over the world. I believe the following makes more sense.

“The real danger, then, is not machines that are more intelligent than we are usurping our role as captains of our destinies. The real danger is basically clueless machines being ceded authority far beyond their competence.’’ (Daniel.C.Dennett, In: The Singularity—an Urban Legend? 2015).


Harm op den Akker et al., 2018. Council of Coaches – A Novel Holistic Behavior Change Coaching Approach. Proceedings of the 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health

Sherry Turkle (2011). Alone together: why we expect more form technology and less from each other. Basic Books, New York, 2011.

Joseph Weizenbaum, 1976. Computer Power and Human Reason: from judgement to calculation. W. H. Freeman & Co. New York, NY, USA, 1976.