BLOG
11 June 2020

Applying State Machines Modelling to Holo4Labs Menu Control

tech

As of half a year, we have been working with HoloLens technology. It is amazing and allows us to interact with the user as we have never done before, through an immersive world using hand interactions.

But, as Uncle Ben once said: “With great power comes great responsibility”.

While this technology is amazing, it can also be problematic. As we can read in HoloLens design doc, the authors admit to limitations in offering ‘specific actionable guidance.’ Much of this is new territory, referred to as ‘lessons we’ve learned’ and ‘avoiding going down that path’.

The HoloLens is a fresh new product with plenty of design guidance and reasonably sophisticated displaying theorem.

For me, the problem with HoloLens is writing on the keyboard where keys can easily become mistyped, wherein I am never sure if my password had been typed correctly.

In this article, we will talk about the state machine, how we did it, and how it solves our problems within the palm menu, which is much simpler to explain. In the second part of the article we will discuss how it prevents accidental clicks by blocking interactions for a specific time after pressing.

The downsides of MRTK plugin

HoloLens applications are developed in Unity Engine. It is a popular multiplatform game engine. Microsoft offers users a plugin called MRKT, which I found to be an excellent piece of code. It can handle writing fast uncomplex apps, allows users to rapidly prototype, and probably has all the other things that you will be searching for.

Hoverer, we found two downsides of using MRTK:

  • We found MRTK not so scalable since many things are done from the Unity scene. Having many dependencies on the Unity scene makes it hard to merge.
  • MRTK is based on MonoBeavhiour, a bridge script between unity and C++. That can cause a variety of problems. In Holo4labs, we have a View – business logic separation and 95% of our scripts are plain C# objects instead of hard to maintain, and with heavy MonoBehaviours. We just wrapped Unity UI controls and implemented MRTK interfaces over it, which gave us more control and flexibility. It is good that much code that MRTK uses is available to programmers. We implemented our own voice system, so it works from code and is easier to maintain in a primarily scaling project.

After those short explanations, we are slowly getting to the core. As our team did cover on buttons, we had many problems with their states and transitions. Therefore, we designed a state machine which is used in many points of the project.

The advantages of state machine modelling

So, what are the profits of this approach?

  1. We can separate a state transitions code. Sometimes this can be quite complex. This arises from any other logic and reacts to the incoming states.
  2. It is relatively easy to draw a state machine based on the initialization of a state machine.
  3. With the right abstractions, things can be used nearly everywhere in your code.

Ok, now let us jump into state machine itself. It is based on ReactiveX which is a library used for event handling. If you have never used it, don’t worry. I’ll try to explain things as simply as I can.

Applying state machine modelling to Palm Menu

In Holo4labs, we have a palm menu which has three states:

  • Closed,
  • Open,
  • and WaitingForClose.

Here, we open the menu when the open palm is detected in front of the user, and close it when it is no longer visible. The biggest problem here was that the hand menu couldn’t be closed immediately. Instead, it has to wait for some time before closing, as detecting open palm during this waiting will abort the timer countdown.

Below, we can see the states initialization and their graph. Formating appears as such.

1 2 3 4 5 6 public HandMenuStateMachine() : base(HandMenuStates.Closed) { availableStateTransitions[HandMenuStates.Closed] = new HashSet<HandMenuStates>() { HandMenuStates.Open }; availableStateTransitions[HandMenuStates.Open] = new HashSet<HandMenuStates>() { HandMenuStates.WaitingForClose }; availableStateTransitions[HandMenuStates.WaitingForClose] = new HashSet<HandMenuStates>() { HandMenuStates.Open, HandMenuStates.Closed }; }

Here, we can see a graph of these state transitions.

Our state machine holds the transition map in the form of:

1 2 Dictionary<T, HashSet<T>> availableStateTransitions;

The dictionary and hash set is collection-optimized for searching by key values where the first key of the dictionary is current state, and where Hash set contains available states to which we can move.

Later, we can write a code on how to handle each state transition. We could also design this code around classes instead of enums, and every transition would contain its own code, but this would be overengineering in my opinion, and be hard to read.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 public override void TryToSetState(HandMenuStates desiredState) { //1. This line of code checks if we can move to desired state if no code just quits if (availableStateTransitions[stateTransitionSubject.Value].Contains(desiredState)) { //2. After checking we notify that the state transition is successful // stateTranzitionSubject.onNext(desiredState) works like Action.Invoke(desiredState) it //is ReactiveX semantic stateTransitionSubject.OnNext(desiredState); //3 When we set state to WaitingForClose we must start our timer if (desiredState == HandMenuStates.WaitingForClose) { //4 we start to listening to timer, waitingDisposable is subscription handle if we call //dispose on it we stop listening (dispose works like Action -= method) //.Delay(TimeSpan.FromSeconds(0.2f)) this is our timer //.Subscribe(_ => TryToSetState((HandMenuStates.Closed))); tells us what to do when timer is //finished it will try move to closed State //if machine will go into open state once more closed state will be rejected waitingDisposable = closeTimer.Delay(TimeSpan.FromSeconds(0.2f)) .Subscribe(_ => TryToSetState((HandMenuStates.Closed))); //5 rise timer event(start timer) closeTimer.OnNext(Unit.Default); } //If during timer counting we go into Open state once more we shall stop listening to our //timer events If (desiredState == HandMenuStates.Open) { waitingDisposable.Dispose(); } } }

Example of palm menu invoking:

1 2 3 public void OpenPalmMenu() { stateMachine.TryToSetState(HandMenuStates.Open); } public void ClosePalmMenu() { stateMachine.TryToSetState(HandMenuStates.WaitingForClose); }

Example of reacting to state changes:

1 2 3 stateMachine.InteractionStateObservable.Where(state => state == HandMenuStates.Open) .Subscribe(_ => screensProvider.PalmMenuViewModel.Open()); stateMachine.InteractionStateObservable.Where(state => state == HandMenuStates.Closed) .Subscribe(_ => screensProvider.PalmMenuViewModel.Close());

Whole Code of HandMenuStateMachine

Base Class:

1 2 3 4 5 6 7 8 public abstract class StateMachine<T> : IStateMachine<T> { protected BehaviorSubject<T> stateTransitionSubject; protected Dictionary<T, HashSet<T>> availableStateTransitions; protected StateMachine(T startingState) { stateTransitionSubject = new BehaviorSubject<T>(startingState); availableStateTransitions = new Dictionary<T, HashSet<T>>(); } public IObservable<T> InteractionStateObservable => stateTransitionSubject; public T PreviousState { get; protected set; } public T CurrentState => stateTransitionSubject.Value; public void Dispose() { stateTransitionSubject.OnCompleted(); stateTransitionSubject.Dispose(); } public abstract void TryToSetState(T desiredState); }

HandMenuStateMachine:

1 2 3 4 5 6 7 8 9 10 11 12 13 public class HandMenuStateMachine : StateMachine<HandMenuStates> { private Subject<Unit> closeTimer = new Subject<Unit>(); private IDisposable waitingDisposable = Disposable.Empty; public HandMenuStateMachine() : base(HandMenuStates.Closed) { availableStateTransitions[HandMenuStates.Closed] = new HashSet<HandMenuStates>() { HandMenuStates.Open }; availableStateTransitions[HandMenuStates.Open] = new HashSet<HandMenuStates>() { HandMenuStates.WaitingForClose }; availableStateTransitions[HandMenuStates.WaitingForClose] = new HashSet<HandMenuStates>() { HandMenuStates.Open, HandMenuStates.Closed }; } public override void TryToSetState(HandMenuStates desiredState) { if (availableStateTransitions[stateTransitionSubject.Value].Contains(desiredState)) { stateTransitionSubject.OnNext(desiredState); if (desiredState == HandMenuStates.Open) { waitingDisposable.Dispose(); } if (desiredState == HandMenuStates.WaitingForClose) { waitingDisposable = closeTimer.Delay(TimeSpan.FromSeconds(0.2f)) .Subscribe(_ => TryToSetState((HandMenuStates.Closed))); closeTimer.OnNext(Unit.Default); } } }