MelodyVR
My role at MelodyVR is lead technical artist, creating content for the MelodyVR platform, pipeline tools for video content and camera technology R&D work.
3D Environments
I’ve created several virtual spaces that serve as video content hubs in VR apps. In order to reach maximum fidelity as a 3DOF environment, these are normally rendered offline in Maya/Arnold and used as 360 images in the VR app. I did modelling/texturing, lighting, rendering, interactive overlays in Unity, scripting for user interactions in Unity.
Video Jumpspot Transition
This is a feature of MelodyVR I created that allows a user to jump from one video to another in a similar location and feel like they are physically moving there. It relies on aligning videos to share the same ‘north’ and calculating 2D texture offset vectors from a 3D point on the sphere surface. These 3D points are manually set using a tool that allows a developer to mark up camera positions by placing markers. The transition effect and the markup tool are both done inside Unity, one in the final VR app, the other inside the editor.
1: The shader allows alignment of two 360 videos via a Y axis rotation. After blending the frames the Y rotation is lerped back to its original position. To achieve the video transition movement, the texture coordinates are offset along a 3D vector to the sphere-surface position of the next camera, which is marked up using a special tool.
2: The shader viewed in 2D. The calculations are done in 3D space inside of the unit sphere, but are mapped back to barrel mapped 2D coordinates.
3: The resulting transition in the video player
User Gaze Tracking
I implemented user gaze tracking into the MelodyVR app to provide insight into how users are viewing content and how we could optimize content based on this data. The data itself was simply logged in regular intervals and stored on a server. I developed a playback engine for this data to allow heat maps, heat spheres, points of interest to all be viewed based on the data of a single performance gathered from multiple users.
Here is an early version of the playback display options, being controlled by a transform in Unity.