MixCast Guides

Guides for getting MixCast up and running!

MixCast™ Studio Setup Guide

Overview

Want to start recording some awesome mixed reality video? MixCast Studio makes setup simple and fast, just follow the three steps below:

  1. Set up your input device
  2. Perform quick setup to align your virtual and physical cameras
  3. Isolate your image with our WildKeyTM background removal.

Now launch your VR title!

 

What You’ll Need

MixCast supports setups ranging from basic beginner streaming to full video production. This guide serves as a starting point for anyone who wants to make mixed reality video.

We assume you're already up and running with your VR headset, have performed room setup and have the ability to launch VR apps. If you're starting from scratch with your Vive or Rift, please refer to the respective manufacturer's instructions for getting started! What you'll need:

  • Camera – this can be a webcam, video camera or any other HDMI camera with an HDMI capture card
  • HTC Vive or Oculus Rift with two controllers
  • VR-ready PC desktop of laptop
  • VR title with the MixCast SDK integrated, or that supports SteamVR’s quadrant mode for mixed reality

oPITe0HnIAv0YQnc-hmds.png

For the highest quality mixed reality video production consider adding:

  • Green screen & lights
  • Vive Tracker or 3rd controller
  • High-quality DSLR or Video Camera

 

Let’s get started...

1. Setting up Your Input Device

Mixed reality requires a real-world camera to put you in the VR scene! MixCast Studio works with a large number of consumer webcams, video cameras and capture cards.

  1. Hook up your camera to your PC and install the necessary drivers
  2. Launch MixCast Studio
  3. Go into “Input device” and select your device from the drop down menu
  4. Select your input resolution from the “input resolution” menu

DFATVTYXrXszfVJh-input.png

Tip: Go into your camera settings and make sure auto exposure, auto white balance, and auto focus are disabled. These will interfere with your chroma key or static subtraction!

Tip: Make sure to disable any on-screen indicators on your camera otherwise your battery indicator could show up in VR!

Note: Cameras will sometimes support higher resolutions but at lower framerates. Experiment with different resolutions to find the best option for your videos.

Note: You can leave the FOV (Field of View) blank for now.

 

2. Align Virtual Camera

MixCast needs to know where your camera is in the real world so it’s aligned with your virtual camera. This is a core part of making mixed reality work and capturing your video from the right perspective in the VR scene.

  1. With MixCast Studio open, put on your VR Headset
  2. Look around to find your virtual camera. Hint - it's probably right in front of you
  3. Point at the “Quick Setup” Button located below the camera and press the Trigger button to start the Quick Setup process. Hint - you can also enter quick setup via a button in the Camera Settings -> Camera Tracking menu
  4. Follow the prompts in VR

Gln3O3YmYZfp6zt1-Untitled-2.png

Note: The first step is the most important – make sure your controller is centered and right up against the lens of your camera.

Tip: If you want more precision in your alignment, select the “+” button instead of the green check mark after hitting the crosshairs. You'll be presented with more crosshairs to further refine your alignment.

Note: The Quick Setup process automatically determines the FOV of your camera.

Warning: If you change the lens or zoom of your camera the FOV will no longer match the virtual camera and you'll need to run Quick Setup again.

 

3. Isolate Your Subject/Player

We need to isolate the player in the video coming from your camera so your background doesn’t show up in your mixed reality video. MixCast Studio offers two ways to do this, which can be combined for an optimal result using our WildKeyTM settings.

Option 1: Chromakeying

If you have a green screen or blue screen your best option is probably Chromakeying. This process removes specific colours, of your choice, from the background. Chromakeying allows you to quickly and easily isolate the player when the background is a different color.

  1. Open the Camera Settings -> WildKeyTM menu
  2. Enable the Chromakeying toggle
  3. Click the Calibrate button
  4. Click the Sample Background button, then anywhere on your feed to remove that color
  5. When you are satisfied with the result, click the Close Calibration button

Tip: Use Ctrl-Z or click the undo button if you accidentally key the wrong colour, or click the Reset button to start from scratch.

Tip: You can adjust the Fade Falloff and Desaturation in the advanced options back in the WildKeyTM menu.

Option 2: Static Subtraction

Static Subtraction works best when your background has more than one colour. This process takes a snapshot of your background (without the subject present) and ‘subtracts’ it from the video. When the subject steps back in front of the camera they appear because they differ in colour and lightness from the background behind them.

  1. Open the Camera Settings -> WildKeyTM menu
  2. Enable the Static Subtraction toggle
  3. Click the Calibrate button
  4. Make sure you are not visible in the scene, and click the Sampling Background button
  5. When you move back into view of your camera you should be visible, minus the background. Click Close Calibration when you are satisfied with the result.

Tip: You can perform multiple samples of static subtraction to improve the quality of the background removal.

Note: Tracked cameras are not supported when using static subtraction.

 

Advanced Features

Below we describe several optional features that can make a big difference in the quality of your mixed reality video.

A. Buffered Mode

Does your video feed lag behind the virtual objects in Studio? Some cameras and setups introduce delay resulting in the mixed reality output lagging behind the virtual scene. If you notice this, enable Buffered Mode to compensate for the delay.

  1. Open the Camera Settings -> Output menu.
  2. Adjust the Video Dealy Compensation slider to a value above 0 milliseconds, until you are happy with the result.
B. Tracked Camera

If you have a third controller or a Vive Tracker you can use it to track your real-world camera. This means you can pick up your camera, move it around and get different angles of the action in real-time!

  1. Ensure the third controller or Vive Tracker is firmly secured to the camera – any looseness can introduce camera shake in your video
  2. Open the Camera Settings -> Camera Tracking menu
  3. Choose your tracked camera device from the dropdown menu or click the Take Closest button.
  4. Now move your camera together with the controller/tracker, and the VR scene will remain in sync with the movement of the camera
C. Enhanced Background Removal

Enhanced Background Removal will automatically ignore any part of the video feed a certain distance away from your hands or head. Since you probably don’t want your lighting rig to show up on-screen, enhanced background removal ensures only the subject/player and their immediate surroundings are displayed.

  1. Open the Camera Settings -> WildKeyTM menu
  2. Enable the Enhanced Background Removal toggle
  3. Adjust the padding so that the player/subject isn’t cropped out, but their surroundings are
D. Player Lighting

SGBOSjJtQVlwrI7P-Lighting.00_00_09_27.Still002.pngPlayer Lighting adds another layer of depth to your mixed reality video by allowing light sources in the VR scene to affect the subject. If you’re carrying a torch through a dark forest, you’ll see the subject illuminated by its light along with the virtual surroundings!

  1. Open the Camera Settings -> Subject Lighting menu.
  2. Adjust Take Lighting – this determines the percentage of all lighting that will affect the player, for instance 0.5 results in half as much light affecting the player. Think of it as a modifier for the entire Player Lighting feature
  3. Adjust the Base Amount – This is the amount of light applied to the video feed by default even if other lights aren’t present. For instance, setting this to 0 will result in the player being completely unlit and dark
  4. Adjust the Light Power – This is how any lights in the scene brighten and color the player

Tip: In MixCast Studio grab the torch to test your lighting settings!

Note: Player Lighting can only be enabled on VR titles with the MixCast SDK integrated. Please contact your favourite developers to enable support for Player Lighting and other great MixCast features!

Launch Your VR Title!

fBrou0PJXsnfNuoB-MixCast_Logo_On.pngIn most cases you'll see the MixCast logo appear in the bottom-right corner of the game running on the desktop - click it to enable the MixCast mixed reality camera. In some cases the developer has chosen to place MixCast options in their own settings menu, so check there if the MixCast icon isn't visible on-screen.

Recording & Streaming

Capture your footage using MixCast recording and streaming options, or use your desktop recording software of choice. We recommend OBS but XSplit, Shadowplay and many others exist so use what's best for your setup.

Questions? Comments?

We'd love to hear from you! Let us know what you think of MixCast, what awesome projects you're working on, and what crazy problems you're trying to solve in mixed reality! You can get in touch with us on -

Thanks for using MixCast!

MixCast™ Capture Guide

Setting up Capture

Sadly not all VR titles support the MixCast SDK yet – you can help by contacting your favourite VR developers and ask them to integrate our simple SDK to enable advanced Mixed Reality features!

However, MixCast Capture, bundled with MixCast Studio, supports Mixed Reality video for any VR title that already supports SteamVR’s built-in quadrant mode (most SteamVR titles built in the Unity engine).

Note: Due to the limitations of the SteamVR Mixed Reality feature only the HTC Vive can currently be used with MixCast Capture.

Note: You will need a third controller to enable SteamVR’s Mixed Reality mode for MixCast Capture to work. Don’t have one? Not to worry, you can create a fake virtual controller, or if you have a Vive Tracker you can trick SteamVR into thinking it’s a controller instead.

 Once you’ve set up MixCast Studio set up (see above) running Capture is simple!

  1. Ensure you’ve selected your third controller as the tracked camera device in Studio
  2. Launch MixCast Capture from Steam or by using the button in MixCast Studio
  3. When MixCast Capture says “Waiting for VR application” launch your desired VR title!
  4. Your VR title will launch, briefly close, and then re-open automatically in Mixed Reality mode!
  5. Bring MixCast Capture back to the foreground (don’t minimize your game) to see your Mixed Reality output

Tip: Make your game full screen by hitting Alt+Enter and then switch to Capture and make it fullscreen as well for best results

MixCast Frequently Asked Questions

What Hardware do I need to get started with MixCast?

At a minimum, you’ll need a fairly powerful PC, a compatible camera (see here), and either the Oculus Rift or HTC Vive headset. To take it to the next level, we recommend setting up a green-screen, and having a third controller or Vive tracker allows you to move the camera while broadcasting.

What kinds of cameras are compatible with MixCast?

MixCast currently supports DirectShow-compatible USB devices, which include almost all webcams as well as some digital cameras and capture cards. 

If your device isn't appearing in the dropdown list, or is, but isn't outputting when selected, let us know, but in the meantime you can try a workaround described here!

Which games support MixCast?

A growing list of games are enabling MixCast broadcasting with the MixCast SDK (look for the MixCast logo in the game’s desktop UI). If your favourite game doesn’t support MixCast yet, let the developer know if you’re interested in using MixCast VR with their product. If you’re working on a title yourself, you can download the free SDK right here!

For many more titles you can use MixCast Capture. For a full list of supported and confirmed MixCast-ready titles, see our App Compatibility page!

Are there keyboard controls for any MixCast Studio functions?

Yes! See the list here.

How do I align the virtual image and camera feed?

This setup step is called Camera Alignment, and is performed while wearing your VR headset with MixCast Studio. 

  1. Find the Camera object floating in the room
  2. Point your controller at the button below the Camera in the middle, labelled Quick Setup, and squeeze the trigger.
  3. Bring the ring of the Vive controller that you used to select the button as close to the physical location of the camera as possible. The output of the camera is shown in VR. You should align the ring to encompass the center crosshairs on screen as largely as possible, center it on screen, and click the Grip button to capture the camera's position. 
  4. Take 1-2 steps back and repeat the following step 4 times:
  5. Hold a controller so its ring is centered in the crosshair presented on-screen. The scale is not important here, just the position. Press the Grip button while in this position.

    You should now have a fully aligned camera feed. Move your controllers around and test that the physical controllers remain overlapping their virtual counterparts as much as possible. If there is a delay between your physical controllers and your virtual controllers aligning, you may want to enable or adjust Buffered Mode.

How do I know when I have a good Camera Alignment result?

You can test your values by standing in the center of the viewable area, facing the Output Display, and holding your controllers in various positions: straight forward, straight out, and straight up (if necessary). If your virtual controllers and real controllers stay overlapping throughout, your alignment settings are correct. NOTE: Your camera’s Field of View value will also affect results.

What can I do if the camera image seems delayed relative to the virtual objects (virtual controllers moving before hands)

Your camera or capture card has “latency”, which can be compensated for by using the Buffered Output mode. The output mode can be set from the Compositing screen in MixCast VR Studio. Once set, you should experiment with the value of the Game Delay field (hit Enter after typing in a value) to find one that causes your hands and controllers to move in sync.

Once Buffered mode is enabled, you can adjust this value in keyboard in the application using the MixCast SDK while MixCast is active by pressing the [ key to decrease the delay of the virtual world, and the ] key to increase it.

How do I determine the Field of View of my camera?

For Mixed Reality cameras: Quick Setup is now the fastest, easiest, and generally most accurate method of determining your camera's Field of View. Put on your VR headset, grab your controllers, and select the button below the floating camera object with the label "Quick Setup" to run through the process.

For Virtual Cameras: Select whichever Field of View gives you yourdesired results! Values can range from 1-179 degrees.

How do I record my MixCast experience to play back or upload later?

Recording application output to video can be handled by whichever screen-capture application you prefer. Some of our favorites are OBS Studio and GeForce Experience.

How do I stream to Facebook Live, Twitch, etc with MixCast?

Streaming to your favorite platform can be handled by whichever screen-capture/streaming application you prefer. Some of our favorites are OBS Studio and GeForce Experience.

What can I do if my 3rd controller is being treated as one of my hands in a VR application?

Try these steps in order until you find success:

  • Hold the Steam button on your hand controllers and select Turn Off Controller, then re-enable it by pressing the Steam button and waving it around until detected
  • Unplug the 3rd controller, wait until your hand controllers appear, and then plug it in again
  • Unplug the 3rd controller, close your VR application and SteamVR, open SteamVR, turn on your hand controllers, then plug in your 3rd controller.

What can I do to improve the quality of static subtraction keying?

Your environment is the biggest factor to the quality of static subtraction keying. Here are some suggestions to try:

  • Disable automatic camera parameter management (auto-focus, auto-light, etc)
  • Check that objects behind you don’t have similar colors to your clothing or skin
  • Ensure that the lighting in the room is unchanging and your movement in the room doesn’t affect the lighting of the room significantly
  • Check if your device’s driver or software has any Automatic image controls enabled. These features will generally interfere with isolation mechanisms, although Auto-Focus can still be enabled for chromakeying in most cases.

What do I need to enable camera movement during filming?

Tracked camera movement involves dedicating one controller (or the Vive Tracker) to the camera you want to move. The controller and the camera should be fastened together so the camera view is unobstructed, as are the tracking sensors of the Vive controller. A rigid bond (such as a 3D printed mount or a mini-tripod clamp) is very important to the accuracy of the tracking over the session. If using a 3rd Controller, it needs to be connected via USB cable. To activate tracked movement in MixCast Studio, AFTER Camera Alignment, open the Camera Placement menu and click the “Take Closest” button. Moving that controller should now update the virtual view.

If your tracking is producing jitter in the position or rotation of the camera, you can enable motion smoothing in the UI. Smoothing can also be used with purely virtual cameras (input device set to NONE) to steady your hand motion.

What does Enhanced BG Removal do and how do I use it?

Player cropping allows you to exclude large areas from the real world camera feed in mixed reality without any additional hardware over your existing setup. This allows for greater flexibility in cinematic shots especially those in motion. It also increases the quality of your other background removal methods by providing additional removal logic.

The player cropping feature can be configured under the Subject menu in MixCast Studio. Player cropping operates by calculating a box around the player using their head, hands, and feet (approximated) positions. As well as enabling this feature, you can configure how much padding to add around your subject's head, hands and feet in Studio.

What does Player Lighting do and how do I use it?

Player Lighting, a feature of MixCast introduced in release 1.4, allows for virtual in-game lights to affect the color of the real player in the mixed reality view. This effect is quite compelling when implemented by developers, and helps blur the seam between virtual and real content. 

To enable Player Lighting, open the Lighting menu in MixCast Studio, and configure the "Take Lighting" slider to have a value higher than 0. Note you can blend this value from ignoring lighting to fully lit. You can then adjust the "Base Amount" and "Light Power" values to create the desired amount of contrast and brightness of the player. Try picking up the Torch object in the scene to test how a small powerful light affects the player.

Once you've configured your values, just jump into your/another application. The player in the mixed reality output should now be receiving lights from the application. If your mixed reality output doesn't appear correct when Player lighting is active in the application, but does in MixCast Studio, first see if the developer has posted any information about the issue. If not, let them know! They may not be aware of the issue or could already be working on it.

What resolution can MixCast output at?

There is no limit to MixCast’s output resolution! Without the need to use split-screen layering MixCast can output at the highest resolution available to your monitor or video capture device for streaming.

Keyboard Shortcuts

These shortcuts can be used in MixCast Studio and any SDK-enabled VR title:

 Toggle MixCast:
Ctrl + m

Hide/Show UI:
~ "tilde"

Hide Controllers
0

Show Controllers
9

Rotate Room
Ctrl + [ rotate left
Ctrl + ] rotate right

 Buffer Time Adjust:
[  +25ms game delay
]  -25ms game delay

Using MixCast Capture without a 3rd Controller

Unlike titles implementing the MixCast SDK, SteamVR requires a third controller to enable quadrant mode (mixed-reality view showing the foreground, background and alpha separately on-screen). As MixCast Capture uses existing SteamVR mixed reality features you will need to use a third controller or the below workaround to use Capture for mixed reality recording.

Thankfully, Second Reality has created controller emulator. This software creates an emulated (virtual) controller that tricks SteamVR into thinking it's a real-world controller. As long as SteamVR shows a third controller in your SteamVR Status panel games should launch correctly with MixCast Capture!

Download the OpenVR-InputEmulator here:
http://secondreality.co.uk/MixedRealityTwoController-0-0-3.zip

Instructions on setting up the emulated controller can be found here:
http://secondreality.co.uk/blog/how-to-create-mixed-reality-videos-for-the-vive-with-two-controllers/

If you have a Vive Tracker you can use it instead of the emulator with this workaround.

Using a Vive Tracker with MixCast Capture

MixCast Capture relies on SteamVR's mixed-reality quadrant mode to capture mixed-reality footage. Enabling quadrant mode requires a third Vive tracked device, but unfortunately many titles don't recognize the Vive Track as a valid tracking device. This depends on the version of the SteamVR SDK used in the title.

Thankfully it is possible to convert your Vive Tracker into a controller by updating its firmware. This is a fast and simple process, but as with any firmware update there's some risk of failure resulting in an unusable device, so proceed at your own risk. That said, this method of changing the Vive Tracker firmware was created by an HTC Vive developer. This process is also reversible.

Once the Vive Tracker shows up as a controller in your SteamVR Status panel you're good to go! SteamVR titles supporting quadrant mode should now launch correctly with MixCast Capture!

Download it here.

Instructions from the original post by the creator:

Vive Tracker Role Changer (current version: 0.8)

VIVE Tracker Role Changer v0.8

 

This tool is for changing the role of the tracker to read in SteamVR as a controller. This is only required for applications that doesn't yet recognize the device correctly, e.g. mixed reality.

 

Directions:

  1. Unplug the Vive from the computer.
  2. Plug in the Tracker via USB cable.
  3. Run tool and follow the prompts.
  4. Tool will report what the tracker is currently reporting itself as and give you the option to switch it's role reporting.
  5. On completion, unplug the tracker and restart SteamVR. Reconnect everything.

Note: While your tracker is reporting itself as a controller, while it shouldn't ask you to update, if it does, do not update the firmware until it's reporting itself as a tracker again.

 

Hopefully I didn't miss anything with the tool. Let us know how it works. Also, if you guys need this tool for another issue outside of the MR issue that we're aware of, please let us know.

If you don't have a Vive Tracker, you can 'fake' a third controller in SteamVR. Learn how here!

Custom Logo Setup

Please note, this feature is only available with paid licenses of MixCast.

Feature

Users can add a custom logo or watermark to their mixed reality videos and snapshots, for branding and copyright protection. This feature is available with the Streamer/Arcade, Education, Professional Film or eSports editions.

A custom logo is most valuable for arcades to brand screenshots taken in arcades, and for streamers to help copyright their mixed reality broadcasts.

What You Need

  • PNG image file with a transparent background

Custom Logo Setup

Place the PNG image in: 

C:\Users\[Name of User]\AppData\LocalLow\Blueprint Reality\MixCast Studio

 After you have placed it, right click on the file and click Properties.