Follow Paulo Pinheiro and his projects on Dev Mesh

INNOVATORS OF TOMORROW YOUTUBE PLAYLIST

INTEL SOFTWARE INNOVATOR PROGRAM

INTEL DEV MESH

SUBSCRIBE TO THE INTEL SOFTWARE YOUTUBE CHANNEL

Welcome to Innovators of Tomorrow. I'm your host Wendy Boswell here to bring you the sights, sounds, and inspirational work from developers in our innovator community from around the globe. Today we are talking to Intel software innovator Paulo Pinheiro about his project Wheelie, a wheelchair that can move using only facial expressions. So let's get this started. 

[MUSIC PLAYING] 

All right so, Paulo, thanks for being here with us today. Can you bring us up to speed on what you've been working on? 

Yeah. I have been working on robotics for my entire career receiving my PhD in the area. And at the time I was working with the iRobot Roomba, the vacuum cleaner robots, to make them faster and to make them smarter. They could plan ahead their cleaning activities considering their residence at home so they could save time and power. 

And then I got part of this software and put it inside a regular wheelchair to make it smarter. A smart wheelchair so the wheelchair could go from point A to point B by itself. And what we discovered was that the users liked the future, but they would like to control their wheelchair and not to be controlled by a wheelchair all the time. So we created a very unique way for the users to control a wheelchair in a more comfortable way. 

Right, so I've seen this device at a couple of events Intel Developer Forum, Paraplegic Olympics, and I believe you call it Wheelie. Can you tell us more about this project? 

Yes. The Wheelie is the first ever computer program capable of translating facial expressions like a kiss, a smile, into comments to control a wheelchair. So quadriplegic people, people diagnosed with ALS, they can now control a wheelchair in a less invasive way, in a more comfortable way. 

This is an amazing project that has the potential to help a lot of people. So, what was your inspiration behind creating Wheelie? 

Well, I was at the airport waiting for my flight, and I saw this girl in a wheelchair. She could not move her hands. She could not move her legs. Her father was helping her to control the wheelchair. And she had a great smile. You know when you see someone who has a great smile, and you cannot stop to thinking about it? It was the case. So I just thought, OK, I don't know how to do this right now. But someone should definitely try to translate that smile into comments to control a wheelchair. 

That is a great story. So you basically took that inspiration, and turned it into a project. Can you tell us how you created Wheelie? 

Yeah. So I got back to the office and the first thing I did was to run idea validation. So I set up a meeting with my quadriplegic people, colleagues, and people diagnosed with ALS. I told them about the idea. I got excellent feedback. We started to write the code, and at that time we had a great machine learning software, but we didn't have the great hardware. Then in Brazil in Sao Paolo, I ran into the Intel RealSense Technology during a [INAUDIBLE] zone that Intel was hosting over there. And boom, we had the right software, right hardware, put all of them together, and went from the idea to the prototype. A few days later, I was given my Postdoc position that I was about to take in Sweden, and we found Hoo Box robotics that is the startup behind the Wheelie. 

So it sounds like you had a team of people that worked on Wheelie with you. Can you tell us more about your team? 

Yeah. Right now we are eight people in the team. And we are all experts in machine learning, artificial intelligence, robotics, and also human behavior analysis. 

Well, it sounds like you cover the gamut as far as everybody brings something to the table. So can you tell us more about the hardware and software that you used in this project? 

Yeah, sure. For the hardware you use a 3D camera. We're using Intel RealSense Technology camera. But the 3D math, the depth map, it's just amazing for our application. It's so precise. I love the design of the camera. It's a reliable products as well. To capture the 3D stream we use the Intel RealSense SDK 2.0. So it's across platforms, so allows us to build prototypes both for Linux and Windows. And then as our on board computer in this latest version we use Intel Nook. So we captured the 3D map, and then we run our software to got the facial end marks. We classify facial expressions, and then we send the comment to a little robot that we place over the joystick of the wheelchair to control the wheelchair. 

That is cool. So Wheelie's gotten a lot of exposure. Can you tell us a little bit more about that? 

Yes, our first global event was the Intel Developer Forum 2016, the IDF. And since then we got a lot of visibility. We got invited to go to the ParaOlympic games in Brazil. Brazil was hosting the Olympic games over there. It was just nice to take Wheelie there so people could use facial expressions to control a wheelchair. And then we have demoed Wheelie in many countries in Europe and all around the US. Our last one was that Johnson and Johnson where we won the prize, and we got invited to be here in the Houston Incubator in the JLABS. 

That's cool. So I think you also published a scientific paper that got quite a bit of recognition around Wheelie. Can you tell us more about that? 

Yeah, sure. At Hoo Box we are all scientists. We are always publishing academic papers. The last one was very important to us because we evaluate the use of facial expressions to drive a wheelchair over a real life scenario. So we asked 30 volunteers to drive a wheelchair using Wheelie facial expressions in a home so they could go through corridors, obstacles, doors. doing turns and displacements in a very precise way. The good thing was that all of them completed the mission. They could do this very well. And they could improve their ability to use facial expressions to drive a wheelchair as well. And also we got some insights about what facial expressions the users like the most to use. Like a kiss, a smile they love to use. Wrinkle nose like this they don't like it. So those kinds of insights are good for our next version of the product. 

So those are the facial expressions that people are most comfortable with? 

Well you have 10 facial expressions. You can choose the ones that you think is more comfortable. You just need five, one to go forward, backward, turn left, turn right, and to stop. But you have 10 to choose to pick it up. 

That's cool that people have given you that kind of input on the facial expressions that work for them. So you mentioned Johnson and Johnson earlier, and I believe you've just joined Johnson and Johnson labs. Can you tell us a little bit more about that? 

Yeah, that has been great. We are here right now at JLABS in Houston at the Johnson and Johnson innovation lab. We're improving the Wheelie in so many different ways. We are improving its product, its design, its usability, safety. We are learning from the experts how we can put the Wheelie's technology on a device, a medical device or consumer device, so it has been amazing. Our latest version we call Wheelie 6 because it only takes six minutes to be installed in any motorized wheelchair available in the market. 

Wow. OK, so this is a kit that can be retrofitted to any wheelchair. 

Yeah, that's it. You don't need a robotic wheelchair, a special wheelchair, just a regular wheelchair, those motorized wheelchairs you have available in the market. 

That is really cool. So is the software open source? Is the community helping to develop it? 

The community is really helping. They are giving us excellent feedback. The software is still learning from them, from their facial expressions. The official recognition software version that works regardless of light condition and position of the head, it's not open source. But the cool thing is that we are creating an open source platform so developers can build their own interface and use this platform to control the robot that is placed over the joystick to control a motorized wheelchair, just a regular one, not a special one. So you just take care about the interface, and the robotic part we are going to take care of. 

That is amazing So, Paulo, what are next steps for Wheelie? Where do you see this going? 

Next steps are amazing. We are getting the technology from the wheelchair input into ICU beds, intensive care units, so we can monitor patients. And so far we can detect the level of pain, we can detect agitation and sedation level. We'll have a detector for spasms and delirium. This is for intensive care units. And two months from now, we are going to embed the technology inside baby monitors so we can track that baby's sleeping records. So it will be just amazing, just capturing facial expressions. 

So for people who are in wheelchairs, or in intensive care units beds, or for patients who are newborn babies, we are capturing facial expressions to detect the human behaviors and predict those risk human behaviors just to make their life better and also save lives. 

That is amazing, Paulo. It is so awesome to see how many people you've already impacted with this invention and how many people that you're going to potentially impact in the future. Thank you so much for coming into the show today and talking with us. 

Thanks for having me on the show, Wendy. 

You can connect with Paulo and learn more about Wheelie at the links provided. That wraps up this installment of the show. Be sure to like this video and subscribe to the Intel Software YouTube channel to keep learning about the innovators of tomorrow. On behalf of an amazing video crew, thanks for tuning in, and we'll see you next time.Welcome to Innovators of Tomorrow. I'm your host Wendy Boswell here to bring you the sights, sounds, and inspirational work from developers in our innovator community from around the globe. Today we are talking to Intel software innovator Paulo Pinheiro about his project Wheelie, a wheelchair that can move using only facial expressions. So let's get this started. 

[MUSIC PLAYING] 

All right so, Paulo, thanks for being here with us today. Can you bring us up to speed on what you've been working on? 

Yeah. I have been working on robotics for my entire career receiving my PhD in the area. And at the time I was working with the iRobot Roomba, the vacuum cleaner robots, to make them faster and to make them smarter. They could plan ahead their cleaning activities considering their residence at home so they could save time and power. 

And then I got part of this software and put it inside a regular wheelchair to make it smarter. A smart wheelchair so the wheelchair could go from point A to point B by itself. And what we discovered was that the users liked the future, but they would like to control their wheelchair and not to be controlled by a wheelchair all the time. So we created a very unique way for the users to control a wheelchair in a more comfortable way. 

Right, so I've seen this device at a couple of events Intel Developer Forum, Paraplegic Olympics, and I believe you call it Wheelie. Can you tell us more about this project? 

Yes. The Wheelie is the first ever computer program capable of translating facial expressions like a kiss, a smile, into comments to control a wheelchair. So quadriplegic people, people diagnosed with ALS, they can now control a wheelchair in a less invasive way, in a more comfortable way. 

This is an amazing project that has the potential to help a lot of people. So, what was your inspiration behind creating Wheelie? 

Well, I was at the airport waiting for my flight, and I saw this girl in a wheelchair. She could not move her hands. She could not move her legs. Her father was helping her to control the wheelchair. And she had a great smile. You know when you see someone who has a great smile, and you cannot stop to thinking about it? It was the case. So I just thought, OK, I don't know how to do this right now. But someone should definitely try to translate that smile into comments to control a wheelchair. 

That is a great story. So you basically took that inspiration, and turned it into a project. Can you tell us how you created Wheelie? 

Yeah. So I got back to the office and the first thing I did was to run idea validation. So I set up a meeting with my quadriplegic people, colleagues, and people diagnosed with ALS. I told them about the idea. I got excellent feedback. We started to write the code, and at that time we had a great machine learning software, but we didn't have the great hardware. Then in Brazil in Sao Paolo, I ran into the Intel RealSense Technology during a [INAUDIBLE] zone that Intel was hosting over there. And boom, we had the right software, right hardware, put all of them together, and went from the idea to the prototype. A few days later, I was given my Postdoc position that I was about to take in Sweden, and we found Hoo Box robotics that is the startup behind the Wheelie. 

So it sounds like you had a team of people that worked on Wheelie with you. Can you tell us more about your team? 

Yeah. Right now we are eight people in the team. And we are all experts in machine learning, artificial intelligence, robotics, and also human behavior analysis. 

Well, it sounds like you cover the gamut as far as everybody brings something to the table. So can you tell us more about the hardware and software that you used in this project? 

Yeah, sure. For the hardware you use a 3D camera. We're using Intel RealSense Technology camera. But the 3D math, the depth map, it's just amazing for our application. It's so precise. I love the design of the camera. It's a reliable products as well. To capture the 3D stream we use the Intel RealSense SDK 2.0. So it's across platforms, so allows us to build prototypes both for Linux and Windows. And then as our on board computer in this latest version we use Intel Nook. So we captured the 3D map, and then we run our software to got the facial end marks. We classify facial expressions, and then we send the comment to a little robot that we place over the joystick of the wheelchair to control the wheelchair. 

That is cool. So Wheelie's gotten a lot of exposure. Can you tell us a little bit more about that? 

Yes, our first global event was the Intel Developer Forum 2016, the IDF. And since then we got a lot of visibility. We got invited to go to the ParaOlympic games in Brazil. Brazil was hosting the Olympic games over there. It was just nice to take Wheelie there so people could use facial expressions to control a wheelchair. And then we have demoed Wheelie in many countries in Europe and all around the US. Our last one was that Johnson and Johnson where we won the prize, and we got invited to be here in the Houston Incubator in the JLABS. 

That's cool. So I think you also published a scientific paper that got quite a bit of recognition around Wheelie. Can you tell us more about that? 

Yeah, sure. At Hoo Box we are all scientists. We are always publishing academic papers. The last one was very important to us because we evaluate the use of facial expressions to drive a wheelchair over a real life scenario. So we asked 30 volunteers to drive a wheelchair using Wheelie facial expressions in a home so they could go through corridors, obstacles, doors. doing turns and displacements in a very precise way. The good thing was that all of them completed the mission. They could do this very well. And they could improve their ability to use facial expressions to drive a wheelchair as well. And also we got some insights about what facial expressions the users like the most to use. Like a kiss, a smile they love to use. Wrinkle nose like this they don't like it. So those kinds of insights are good for our next version of the product. 

So those are the facial expressions that people are most comfortable with? 

Well you have 10 facial expressions. You can choose the ones that you think is more comfortable. You just need five, one to go forward, backward, turn left, turn right, and to stop. But you have 10 to choose to pick it up. 

That's cool that people have given you that kind of input on the facial expressions that work for them. So you mentioned Johnson and Johnson earlier, and I believe you've just joined Johnson and Johnson labs. Can you tell us a little bit more about that? 

Yeah, that has been great. We are here right now at JLABS in Houston at the Johnson and Johnson innovation lab. We're improving the Wheelie in so many different ways. We are improving its product, its design, its usability, safety. We are learning from the experts how we can put the Wheelie's technology on a device, a medical device or consumer device, so it has been amazing. Our latest version we call Wheelie 6 because it only takes six minutes to be installed in any motorized wheelchair available in the market. 

Wow. OK, so this is a kit that can be retrofitted to any wheelchair. 

Yeah, that's it. You don't need a robotic wheelchair, a special wheelchair, just a regular wheelchair, those motorized wheelchairs you have available in the market. 

That is really cool. So is the software open source? Is the community helping to develop it? 

The community is really helping. They are giving us excellent feedback. The software is still learning from them, from their facial expressions. The official recognition software version that works regardless of light condition and position of the head, it's not open source. But the cool thing is that we are creating an open source platform so developers can build their own interface and use this platform to control the robot that is placed over the joystick to control a motorized wheelchair, just a regular one, not a special one. So you just take care about the interface, and the robotic part we are going to take care of. 

That is amazing So, Paulo, what are next steps for Wheelie? Where do you see this going? 

Next steps are amazing. We are getting the technology from the wheelchair input into ICU beds, intensive care units, so we can monitor patients. And so far we can detect the level of pain, we can detect agitation and sedation level. We'll have a detector for spasms and delirium. This is for intensive care units. And two months from now, we are going to embed the technology inside baby monitors so we can track that baby's sleeping records. So it will be just amazing, just capturing facial expressions. 

So for people who are in wheelchairs, or in intensive care units beds, or for patients who are newborn babies, we are capturing facial expressions to detect the human behaviors and predict those risk human behaviors just to make their life better and also save lives. 

That is amazing, Paulo. It is so awesome to see how many people you've already impacted with this invention and how many people that you're going to potentially impact in the future. Thank you so much for coming into the show today and talking with us. 

Thanks for having me on the show, Wendy. 

You can connect with Paulo and learn more about Wheelie at the links provided. That wraps up this installment of the show. Be sure to like this video and subscribe to the Intel Software YouTube channel to keep learning about the innovators of tomorrow. On behalf of an amazing video crew, thanks for tuning in, and we'll see you next time.