Symbol Recognizer VR

WARNING: 5.1.0 engine version has issues with widget intaractions for VR input (https://issues.unrealengine.com/issue/UE-166377). The issue is FIXED in 5.1.1 version.

The tool provides an environment for recording patterns that can be recognized during game, e.g. unlock doors or cast a spell if the drawing is accurate enough.

Showcase Video

Demo Project Video: Link

Tutorial Video: Playlist - 1. Editor, 2. Runtime, 3. Game

Demo Project: Link

The plugin is divided into four sections that guide users through the process of creating a neural network:

1. Profile - This section allows users to create an asset to store their neural network and settings.
2. Record - Users can record 2D/3D inputs to train their neural network with the help of this simple tool.
3. Train - This section enables users to feed their neural network with previously gathered input data.
4. Test - Users can select a trained neural network and draw a pattern to test its accuracy against 'symbols' that were fed into the network.

Once a profile with working neural network is complete, users can do a simple gameplay setup in blueprints or code. The idea is to record the user's input in-game as Vector2D for mouse or Vector3D for motion controller, and then identify which learned pattern matches the user's drawing.

All of the gameplay functionality is exposed to the blueprints by a global object 'SrVRRecognizer' and a basic implementation may follow those steps:

1. Grab SrVRRecognizer and call the function 'StartRecording' when the TRIGGER button is pressed.
2. Check a flag 'IsRecording' in Tick and if true, call 'RecordData' passing motion controller world location.
3. Call 'StopRecording' when TRIGGER button is released.
4. Call 'GetAccuracyList' once recording has stopped, which gives ratio (0-1) for each trained symbol. If the accuracy for an expected symbol is high enough (close to 1), then cast a spell (spawn emitter?)
5. Call ResetRecording once you are done with current recording and want to do a test for a new one.

Note: Even though there are simple examples in the tool showing how to paint in 3D using Niagara or 2D onto a mesh with CanvasRenderTarget, the plugin is not about visualizing a drawing effect in-game and it's up to the user to add any visual effects if necessary. The core feature is about recording player's input as Vector 2D/3D and testing it against trained neural network.