Ni - Labs

Ni - Labs :

Project ( Computer Vision | MR-Driving )
プロジェクト ( 画像認識 | 複合現実ドライビング )
项目 ( 图像识别 | 混合现实模拟驾驶 )

Catalog | カタログ | 目录

①. Teleoperation of Robot Arm with Haptic Feedback [Team work]
       触覚フィードバックを用いたロボットアームの遠隔操作
       具备触觉反馈的机器人机械臂远程操作
②. MR Driving
       複合現実ドライビング
       混合现实驾驶模拟

①. Teleoperation of Robot Arm with Haptic Feedback [Team work]
触覚フィードバックを用いたロボットアームの遠隔操作
具备触觉反馈的机器人机械臂远程操作

1. Overview / 概要

When people grasping an object with hand, they can tell the object with the tactile feedback, even closing their eyes. In this project, we will challenge the possibility to teleoperate a robot arm with the haptic sensors on both robot hands and human hands. And to verify the whether the close loop feedback between the robot and human can essentially improve the teleoperation efficiency.
手では物体を把持する際、目を閉じていても触覚フィードバックによって物体を認識することができます。本プロジェクトでは、人の手とロボットハンドの両方に触覚センサーを装着し、ハプティック情報を用いてロボットアームを遠隔操作する可能性に挑戦します。さらに、人とロボット間のクローズドループフィードバックが遠隔操作の効率を本質的に向上させるかどうかを検証します。
当人用手抓取物体时，即使闭上眼睛，也能通过触觉反馈来判断物体。本项目中，我们尝试使用安装在机械手和人手上的触觉传感器，实现机器人手臂的远程操作。同时，我们希望验证机器人与人之间的闭环反馈机制，是否能够从根本上提高远程操作的效率。

2. My Work / 担当

This is a team project. My responsibility is to utilize an RealSense camera to perform real-time tracking of both the robotic arm and the target object to be grasped.
本プロジェクトはチーム開発であり、私は RealSense カメラを用いて、ロボットアームおよび把持対象物のリアルタイム追跡を担当しました。
这是一个团队项目，我负责利用 RealSense 摄像头实时追踪机械手臂和需要被抓取的目标物体。
Device / 設備:
Intel RealSense Camera D435i

②. MR Driving
複合現実ドライビング
混合现实驾驶模拟

1. Overview / 概要

This project is a mixed reality driving simulation built using the Oculus Quest 3, the Logitech Gran Turismo Force steering wheel, and the software Live for Speed (LFS) and Reality Mixer. Users can operate the virtual car in the VR environment with a real steering wheel, and during the experience, they can see their actual hands and the physical controller. This setup provides a near-realistic experience of driving a car on the road.
本プロジェクトは、Oculus Quest 3、Logitech Gran Turismo Force ハンコン、そして Live for Speed（LFS）および Reality Mixer を用いて構築した混合現実（MR）型のドライビングシミュレーションです。ユーザーは実際のハンドルを使って VR 内の車を操作でき、さらに体験中は自分の本物の手やコントローラーが視野に表示されます。これにより、まるで実際の道路を運転しているかのような、非常にリアルに近い走行体験を実現しています。
这个项目是利用设备Oculus Quest 3和Logitech Gran Turismo Force以及软件Live for Speed (LFS) 和Reality Mixer构建的混合现实驾驶模拟体验，用户可以使用真实的方向盘控制VR游戏中的虚拟汽车，在体验中用户可以看见自己真实的手和控制器，近乎现实地在公路上驾驶汽车。

2. About Reality Mixer

Reality Mixer lets you bring real objects into VR. It uses the cameras on the headset to show a real object and its surroundings in the current VR environment. It can be used at the same time as other VR software and games.
Think of objects like keyboard & mouse, a racing wheel, HOTAS, a clock on the wall, a drink on your desk, a phone, a tablet, a chair, etc. Also larger real objects can be used. Like the door to your room, so you can see who comes in, or the couch with your friends.
Bringing real objects into a virtual environment is known as 'Augmented Virtuality'.
Reality Mixer creates 'boxes of reality' which are shown in VR.
Apart from these 'camera boxes' the app also provides a mixed reality Cockpit for use with simracing and flight simulators, a Floor Viewer to quickly see the real floor around your feet and a Lap Viewer for use with seated experiences such as Big Screen. Lap Viewer can be used for quick glances at a normal (untracked) keyboard on your lap.
If you have a Vive or Tundra Tracker, you can attach a camera box to it so the tracked object will be visible wherever it is in your play area.
The real objects (and their immediate surroundings) don't have to be visible all the time. There are three activation methods:
- Activate by Proximity: Objects (camera boxes) will fade out when you are too far away. This fade out distance can be set by yourself.
- Activate by Gaze Angle: Look straight at a camera box to make it visible. Use this activation method with seated games where you have things on a table next to your seat. The activation angle and optional activation delay can be configured. Using an activation delay means you have to look at the object for a second before it starts to show itself, to prevent accidental activation. Floor Viewer and Lap Viewer both use Gaze Angle Activation by default, but can also use Proximity or ButtonPress activation.
- Activate by Button Press: Press a keyboard, mouse, gamepad, HOTAS or racing wheel button to turn on/off a camera box. When an object is not visible, it will not use any resources.
If you have an Elgato Stream Deck, or if you use Touch Portal on your phone or tablet, you can use it in VR with Reality Mixer.
Chroma keying (green screen) makes the background of the objects transparent so the VR world shows through. This can be useful for objects such as a racing wheel and HOTAS.