>> Devin Boyle: Let me introduce the next person. I’d like to introduce our next speaker dr., Pilar Orero who is being interviewed on the topic of research on accessible xr, I’ll leave it to you.
 >> Shiri Azenkot: Thanks, Devin, that was a really great interview, and thank you, Joel. So Pilar, it's nice to see you again and get a chance to speak with you.
 >> Pilar Orero: yes, thank you for asking me, for inviting me to speak here. I’m very, very -- yeah.
 >> Shiri: We thought of you right away because you've done some really great research on making xr and specifically 360 video accessible. So as a starting point, i know that you've been involved in three major projects in this area, so could you give a brief overview of these projects and their main goals and also tell us how they relate to one another?
 >> Pilar: okay. Four years ago we started thinking how would accessibility services, the main services that would be subscription, subtyping or captioning, how would they be displayed on 360. So we've requested some -- in the European commission, and they gave it to us. And that was fantastic because that allowed us to get together people from across from different countries, mainly Germany, the uk, and Spain, and try to find solutions to how would the subtitles be displayed, how would the transcription be done in 360 and how sign language would be also displayed. So that was what we've been working very hard in the past three years. Now it's left a legacy of a media player for accessibility services, it left a legacy of two tested subtitle display modes. That one was -- and also left a legacy of our understanding that the many challenges of sound in 360. We thought that the challenges would be in the visuals of 360. When we went to the research, we realized that actually the sound is something that was very interesting, and we haven't solved the problem.
 >> Shiri: So you described the first project that produced the player, right?
 >> Pilar: Yes.
 >> Shiri: And what's that project called?
 >> Pilar: That was called IMAC. IMAC is when we developed the player. We developed the editor to generate that type of sign language, and we also did the research about how to -- where to display them and testing that with end users, and getting the -- how they were from user requirement, and then what are we doing now is to apply whatever we learned from the previous project into two projects. One is an opera, so we're making opera accessible 360 with VR. Why? Because we need to -- this project is what is paying for is for opera as a social interaction for people in difficult situations. For example, in may in the jail in Portugal very young people in jail in Portugal what we develop activity with opera, and then they get the opera through VR. Same with island, for example. The challenge there is it's a country that's very rural. It's very difficult for people to go to the opera in Dublin. So what we're doing there is VR to get these immersion. So yeah, that's what we're doing. We're just applying what we've learned in this to one to do with opera.
 >> Shiri: Oh, okay. I love that.
 >> Pilar: It's quite interesting actually, why migration, because apparently -- because migration has used VR as the media to portray the issue of migration. The "New York Times" on all -- the BBC all these mayor media outlets. They've used VR for the topic for the theme of migration.
 >> Shiri: The first project produced the player. The second project is using the player to make opera more accessible to different audiences, and then tell me a little more about migration and where VR and accessible VR plays a role.
 >> Pilar: Okay. Migration is to do with -- this project is a bit daunting. In the 20th century, Europe there have been a lot of internal wars, a lot of internal migration within Europe. If you think about the holocaust and the polish concentration camps, for example, all the Italian. We've had -- or the Spanish civil war, for example. So we've had -- we all think, we like to think that migration is from outside to inside, so i think in the U.S. you believe that it comes from South America to the border with Mexico. We believe in Europe, popularly we believe that people are coming from cdl. They're coming from Pakistan, and they're coming from the west, from the east to the west, the border being in turkey. That's the perception that we have, but we don't believe, we don't think hard enough. We don't realize that you have Europe in itself has been a long -- has a long tradition of bringing us the barbaric acts between ourselves. So this project is taking back to reflect on the idea that not all the migration comes from the east to the west because it is within Europe and because migration has always been in the last few years 360, and there's so much footage, so much content to put a facility on it. Because we have the languages and we need the captions.
 >> Shiri: Right, i understand how accessibility with people with disability was the main focus in the IMAC player, but then are people with disabilities the focus in the opera and migration project also?
 >> Pilar: No, not at all, but no, the idea there is not for people with disabilities. It's for everyone, so we just opened the idea that if anybody would need an alternative communication. Because in Europe we have -- we speak over 200 languages. So research here, it takes much longer because the end user is never just a single language end user and the media that we analyze is never one language. It's multiple language. So communication as a barrier or as a disability, is an outlier for us in Europe. So we need accessibility as a given, as a given asset. Otherwise we don't understand each other.
 >> Shiri: Right, right, so you see accessibility for people with disabilities as just one facet of making something accessible to the population at large.
 >> Pilar: I think the way you perceive a disability in the U.S. as people with disabilities, that is just one that -- in here because shared language is a massive step, so it's not the same to be a disabled person in Spanish or in English as being a disabled person in Slovenian, which is a very small language. Language technologies to go with it. So it is an added -- an added problem, an added barrier to be a disabled person. In Slovenia as being a disabled person in a very large language.
 >> Shiri: That makes a lot of sense. Let's get back to the player. I think that's a really important contribution that you and your team have made. I want to ask you a few more questions about it. From what i understand, so this player allows people, content creators, to add features that make the videos fully accessible, and people can then watch these fully accessible 360 videos in the player, right?
 >> Pilar: Yes.
 >> Shiri: So what were some of the key research questions or challenges that you came across when you were designing the player?
 >> Pilar: One of the key issues -- there is no -- i think it was tom. There is no standardized mechanism to switch on and off all the services. We still don't know how to start the captions, how to start -- this was very interesting on how we came out was just to put the head down and then activate. So we were trying to understand with the use of requirements there all the time, how would -- how are we going to activate whatever services we wanted when you don't have the hand -- when you’re in the -- and then you need to choose from the screen what do you want to choose when you don't see the screen or when you can't -- so alternative ways of interaction with the screen, how to activate and interact with the menu was the biggest challenge for us.
>> Shiri: And there are lots of different devices that you can watch 360 on. And those different devices have different input modalities. Which ones did you focus on.
 >> Pilar: We only looked at, again, videos, which had mounted displays, where did the oculus go, and then for us was to, one, activate them, and two, try to find out what's the usability, try to do usability tests to make the easiest way to activate multiple languages. For example, you would need to activate what language you want the content in being in Europe, language in easy to read, language in audio subtitles. So it was like it's not just one service that you have to activate. There are multiple services in Europe that you have to activate in order to produce the content properly.
 >> Shiri: Right, so what kinds of modalities did you experiment with, or which ones did you find worked best?
 >> Pilar: Well, we experimented for captioning, audio description, sign language, and audio subtitling, something i don't think you do very much. We do it in Europe, and when you have captions and you have a voice reading the captions aloud. That's very good when you are watching a movie in Japanese and you can't read the subtitles in English because of the distance, for example, and because you can't read English or because you will get tired of it in English. You have this voiceover. We also did easy to read as well.
 >> Shiri: yeah, i wish we had more of the audio subtitling in English. But can you tell me what kinds of modalities you looked at, like you mentioned that you looked -- there was a head gesture for activating the captions, for example. Can you tell me about what kinds of input you explored and which ones you ended up using?
 >> Pilar: Okay, the other one. We used the head. The other thing that we used was a google home, so again, we could activate and we could interact with any of the services with voice command, taken only in English because the voice commands were not allowed or available in any other languages so, again, that was one that worked really well, was fantastic. It wouldn't be a good idea in Europe unless -- i keep on going on these languages. But believe me, it's quite frustrating when the only language that is developed in English and one language is not the only language we have in Europe, so it's sort of crashing for us. So I’m sorry to be so boring. That's the best. That's the first barrier we find in Europe. On top of all the disabilities, of course.
 >> Shiri: Right.
 >> Pilar: Captioning was the one that we had best -- the most success, i think. But then at the same time, it was something that we had to very quickly set for two modalities, and this is something where we're going now. Our team is develop these prototypes. I think i sent you a link of what we're doing, and we settled very fast but only two modes, and where we're testing now is to find different ways of presenting captions, and for example, when two people speak. When we did all the tests up to now, we only have one person speaking, just two lines of captions and -- the traditional way. How about when two people -- how about when three people are talking, that sort of thing is what we're now trying to understand, also understand when you can talk or happen in the action to have more time to read the captions or to have more time to listen to the description because i think one very important requirement from the use that we've realized is that using or watching -- is not the same as watching 2d or 3d. You do spend more time, things that it is random, it is quite random the way you search around. So that is something that we've learned and has made everything more complex and more interesting.
 >> Shiri: So i wish we had more time. I know that you've done a lot of user studies and interacted with people with disabilities directly, but let's see if we have some questions from a stock participant, Jesse, any questions?
 >> Moderator:  No questions. We've got a couple people typing.
 >> Shiri: Okay. In that case – I’ve got one now. Okay, thank you. So i have about including more researchers for disabilities and cultural backgrounds to tackle some caption design challenges.
 >> Pilar: This is basic. Though it is very sad, i have to say, we've been doing tests with captions with over 4,000 people from different cultures from Europe, and people like what they're used to. People still say they like two lines under the face, and that's disappointing. But that's the way it comes up time and time again. What can we do?
 >> Shiri: Any other questions?
 >> Moderator: We have another question, so for audio subtitling, how much accuracy is needed in entertainment content for acting versus just reading the subtitles?
 >> Pilar: To read the subtitle, to read the captions, you don't need any access. You just use a text to speak engine and out it goes. It depends on how good or how bad the text to speak engine is. You will get more sentiments, more intonation or less flat delivery, but there is no -- i mean, if the captions are good, then the -- what you get is good, is so cheap as well. I don't know why it's not used more popularly or more across everywhere. I think it's a great service.
 >> Shiri: It should be. It would be a great accessibility feature for people with low vision. I would personally love to have it. Any other questions? We have time for one more. One more short question.
 >> Moderator: Wait, sorry one more question. Just coming in. Beyond captioning, has there been in also providing audio description as well for these environments?
 >> Pilar: Yes, it has. We've very nicely we thought auto description would be just the usual one voice like what we describe the voice of god as we used to. Now the sound, you have directionality and you have distance as well. And that gives a lot of information about where the sound is produced, so that opens up the number of different types of a description that could be offered, for example, taken where the sound is produced or that sort of thing, so in fact, we started doing research, but there is so much more to do and so interest with auto description and sound, disability in sound in 3d is wonderful. It's really nice area of research.
 >> Shiri: Yeah. Well, i wish we had more time, but we do need to wrap up, so do you want to give anyone your website or the website for the projects so they can go to them for more information?
 >> Pilar: Yes, i think I’ve sent it to you. If not you can find in my bio note, my email address. You can write to me, and I’ll right to you straight away.
 >> Shiri: Right, but do you want to say it just for the audience to have?
 >> Pilar: My email address is Pilar.Orero@uab.cat. Yeah, any questions, please send me a line and I’ll send them to you.
 >> Shiri: Great, and you can also google Pilar's name and you'll come across her research and publications.
 >> Pilar: Great, thank you.
>> Shiri: this has been really interesting. And so many open questions left to tackle Larry, i think we have a break first, don't we?
 >> Larry Goldberg: yes, during the break there will be an additional demos available for people to look at for the next 20 minutes if you haven't already checked out the demo videos, you can go look at those now and head over to slack to ask questions of the presenters. We'll be back here in 20 minutes, and Shiri will be presenting some of the research and demos as well.