User Interface Programming for Games

USER INTERFACE PROGRAMMING

FOR GAMES

Luis Sempé

Copyright © 2014 by Luis Sempé

All rights reserved. No part of the material protected by this copyright may be reproduced or utilized in any form, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the copyright owner.

ISBN-13: 978-0-9921491-4-7

Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

To Gaby, the most beautiful and loving human being, you have given me such wondrous gifts.

To Luka, Joseph, Emma and Ethan, you are my source of inspiration and wonder, the sky is not the limit!

Contents

1 Mathematics for Games

1.1     Notation

1.2     Linear Algebra

1.2.1     Vectors

1.2.2     Matrices

1.2.3     Positions

1.3     Rotation

1.3.1     Rotation Demystified

1.3.2     Object Rotation

1.3.3     Rotation in 3D

1.3.4     Translation

1.3.5     Wrapping up Rotations

1.3.6     Scale

1.3.7     Matrix Concatenation

1.3.8     Euler Angles

1.3.9     Quaternions

1.4     Geometry

1.4.1     Shapes

1.5     Alignment

1.5.1      Alignment of Two Rectangles

2 User Interface Models

2.1     Retained Mode

2.1.1     Callbacks

2.2     Immediate Mode

2.2.1     Polling

2.2.2     Rendering Passes

2.3     Diegetic Interfaces

2.3.1     Render To Texture

2.4     Augmented Reality

3 Rendering

3.1     Viewports

3.2     Title Safe Area

3.3     Project (World to Screen)

3.4     Unproject (Screen to World)

3.5     Picking

3.6     Vertex Buffers

3.7     Index Buffers

3.8     Culling

3.9     Color

3.10     Blend Modes

3.10.1     Opaque

3.10.2     Alpha

3.10.3     Additive

3.10.4     Multiplicative

3.10.5     Screen

3.11     Programmable Rendering Pipeline

3.11.1     Vertex Shaders

3.11.2     Pixel Shaders

3.12     Rendering Primitives

3.12.1     Quads

3.12.2     Cube

3.12.3     Pyramid / Cone

3.12.4     Ellipsoid / Sphere

3.12.5     Matrix Axes

3.12.6     World Space Text

3.13     Binary Space Packing

3.14     DirectXTK

3.14.1     SpriteBatch

3.14.2     SpriteFont

3.15     Cinder

4 User Interface Essentials

4.1     Sprites

4.1.1     Animation

4.1.2     Sprite Sheets

4.2     Cameras

4.2.1     Projection

4.2.2     View

4.2.3     World

4.2.4     Camera Interface

4.3     Controls

4.3.1     Properties

4.3.2     Interface

4.3.3     Container

4.3.4     Labels

4.3.5     Buttons

4.3.6     Text Boxes

4.3.7     Message Box

4.3.8     List Box

4.3.9     Scroll Bar

4.3.10     Dropdown Box

4.3.11     Slider

4.3.12     Menus

4.4     User Interface Elements

5 Heads Up Display – HUD

5.1     Health Bar

5.2     Progress Bar

5.3     Digital Displays

5.4     Damage Indicators

5.5     World Space Markers

5.6     Grids

5.6.1     Drag and Drop

5.7     Mini Maps & Radars

5.8     Optimization Opportunities

5.9     Analog Gauges

5.10     Subtitles

5.11     Development Tools

5.11.1     Development Console

5.12     Heat Map

5.13     Gradients

5.13.1     Linear Gradient

5.13.2     Radial Gradient

6 Useful Patterns & Techniques

6.1     Enum Flags

6.1.1     AND Operator &

6.1.2     OR Operator |

6.1.3     XOR Operator ^

6.1.4     Inversion Operator ~

6.1.5     Left Shift <<

6.1.6     Right Shift >>

6.2     Beware of Singletons

6.3     Model View Controller (MVC)

7 Advanced Topics

7.1     Animation with Interpolation Curves

7.1.1     Ease Functions

7.1.2     Ease Curves

7.1.3     Interpolator

7.1.4     Animations

7.2     Transformation Hierarchies

7.3     Virtual Reality

7.3.1     Stereo Rendering

7.3.2     Distortion Correction

8 Localization

8.1     Text Localization

8.2     Image Localization

8.3     Audio Localization

8.4     Asset Localization

8.5     Cinematic Localization

8.6     Localize Early

Conclusion

Foreword

More than ever, gamers these days can be fickle and impatient. If the game you just spent months or years creating doesn't boot up almost instantly, and allow the player to jump right into the action, his or her interest may be lost, and your game might never be turned on again. The player's experience with your game starts---and ends---with your game's user interface. It is the first thing the player sees when the game starts. And it is the primary means by which the player interacts with your game's universe. It is also one of the primary means by which the player judges the quality of your game as a whole. As such, more than for any other kind of software product, games are expected to have incredibly intuitive, effective, and attractive user interfaces.

Certainly many educational resources exist to aid an aspiring user interface programmer. However, these resources are almost entirely aimed at developers of desktop, web and mobile apps. Sadly, very few educational materials are targeted specifically toward the creation of graphical user interfaces for interactive entertainment.

Thankfully for game programmers everywhere, Luis Sempé has taken it upon himself to rectify the situation. The book you see before you fills this gaping hole in game programming pedagogy, and does so extremely well. User Interface Programming for Games covers all the key topics, from laying all of the necessary foundations, all the way to illustrating how to put the finishing touches on your game's UI. And it does so in an accessible and easy-to-understand manner.

This is no surprise to me, given Luis' excellent track record in the games industry. I first met Luis when we worked together on  Medal of Honor: Pacific Assault  (MOHPA) at Electronic Arts, Los Angeles. Even during these relatively early days of his career, Luis always brought with him an infectious enthusiasm and a thirst for learning, as well as a keen knack for solving problems and a strong sense of professionalism and follow-through. I recall being particularly impressed with Luis' game development skills when he implemented an entire flight-based mini-game for MOHPA, including the aircraft's diegetic user interface (i.e., a user interface which exists within the game world, rather than being overlaid in 2D over top of the 3D display). He also worked on numerous other tools and engine systems for MOHPA, including implementing various runtime visual effects and helping to develop a powerful terrain editor for the team.

Since then, Luis has continued to amass a great deal of expertise, both as a game programmer in general, and in the art of creating graphical user interfaces for games in particular. He was in charge of the UI in Army of Two at Electronic Arts, Montreal, and handled the core UI technology for Deus Ex Human Revolution at Eidos, Montreal. He now has ten years of experience in game programming, and numerous commercially-successful games under his belt. I'm confident you'll see Luis' experience and expertise shining through as you read this book.

It is a great time to be learning the craft of game development. A wide range of tools are available to you, as you undertake this exciting educational journey: from online tutorials to sample code to a host of articles and books. But until now, information on user interface programming for games had been sorely lacking. The book you have in front of you expertly fills this gap, and it will undoubtedly become an invaluable tool in your game programming toolkit.

Jason Gregory

Lead Programmer, Naughty Dog Inc.

Author of Game Engine Architecture

www.gameenginebook.com

About the Author

Luis Sempé has had the privilege of developing AAA games professionally as a software engineer for the past ten years. His passion for game development has driven him to develop expertise in many disciplines. During his career he has developed systems and features in 3D graphics, created gameplay features and authoring tools, developed artificial intelligence systems to simulate flight combat and character behaviors, developed user interface systems for many games and integrated, supported and extended user interface middleware. He has also developed systems for video streaming, localization, and general game engine and tools.

User interface development has been a staple of his career, designing and implementing user interface systems for Medal of Honor: Pacific Assault , Army of Two and in particular, the critically acclaimed Deus Ex Human Revolution.

In the June/July 2010 issue of Game Developer Magazine he shared his design for the technology that drives the conversation system and character social interactions in Deus Ex Human Revolution in the article Machina Ex – Creating a State-driven gameplay system for Deus Ex: Human Revolution .

When he is not developing games professionally he is researching and prototyping game features in C++, XNA, Unity and any new technologies that seem interesting. He has a deep appreciation for independent and hobbyist games and sometimes participates in the indie games community discussions.

About the Code

The goal of this book is to provide you with a good conceptual and practical understanding of user interface programming applied to games.  Most examples provide code that works out of the box, however, it is impossible to anticipate every use case, readers are encouraged to use the code as a starting point towards developing their own UI systems or to implement their own features.

Code organization is a complex topic in itself within a project’s structure, presenting code on paper has its own challenges, for example, in C++ we often write code such that the declarations exist in a header file (.h), while the actual operating code will be written in a code file (.cpp).  The examples in print use a more concise presentation that highlights the relevant parts of the code and may not always reflect the same organization as it is in the live code repository.

In many cases, this book will use the simplest implementation of many data structures with the goal of illustrating the concepts and use cases without delving into complexity that may result from code that should generally be optimized, when appropriate code optimization suggestions are given.

Many of the classes and data structures used in this book have been made deliberately verbose with the intention that if you are working on an existing or proprietary code base, you should be able to find the respective objects or functions within your own technology.

The coding style is meant to be simple and concise, some choices are practical for illustration purposes, while others are subjective.

The code is presented in C++11 and the examples use Direct3D 11 and the DirectX Toolkit. http://directxtk.codeplex.com/

For more information about the source code please visit http://uipg.sempemedia.com

Preface

Style matters.

The development of fluid, visually pleasing and efficient user interfaces is a complex endeavor. The goal is to find an elegant way to display useful and time critical information to players in a way that is intuitive. Players need to make split second decisions without thought or hesitation. A good user interface will become an indispensable aid.

As software developers, our instinct is to create and develop something that works, and we stop at nothing until it does. Once our feature works, we feel excited and we want to show off our accomplishment to the world.

But the world may not be ready to see our creation, more pointedly, our creation may not be yet ready to be seen. Style matters.

When we develop a feature, before we share it with the team, before we submit it into version control and commit it into the project’s history, we need to be confident not only that it works, but that it looks good, good enough to be understood and appreciated. Spend the extra time to make sure the user interface elements are properly aligned, scaled, colored. Make certain that text fields are in the right place, font sizes are correct. Make screens and images fade, rather than pop.

When developing a user interface, everything we do will be seen first by the development team. Just as players eventually will, developers rely on the user interface to gain more information about the game world as they develop their own features.

Strive to make the gap from placeholder to production quality as small as possible, be mindful of the art direction and make placeholders conform to the style and colors set by the artists. Attention to detail and a working and visually pleasing user interface will make a significant difference during the game’s production.

Acknowledgements

Writing this book was a fun and exciting experience and I owe much of it to my family. My wonderful wife Gaby who always motivated me to keep going, always bought me those few extra minutes I needed for writing or developing the sample code during the day or night and even proofreading portions to make sure I wasn’t rambling on. I am also very grateful to my kids Luka and Joseph whose excitement and curiosity has been a constant source of fuel for my motivation. And to my twin babies, Emma and Ethan whom I often held in my arms while I scribbled down notes, algorithms and ideas for this book. And to my mom Mirtha and my dad Jorge who raised me around computers and bought me my first game programming book when I was 14 years old, I have not stopped making games since!

The inspiration from this book was born during my time at Eidos Montréal where I had the distinct pleasure of creating Deus Ex Human Revolution alongside many talented engineers and some of the most creative and passionate artists, musicians and writers I have ever met. I learned so much from every single one of them and much of this book was written with them in mind. In particular I am grateful to Daniel Fortier for his patience and trust, many chapters of this book were born from conversations we had or problems we faced and how we solved them. He showed me that programmer art does not mean it has to be bad art, style matters. I am also grateful to Alexandre Richer and Fernando Secco, together, the four of us developed the vast majority of the UI for DHXR. I could probably fill a few chapters with anecdotes of our often spirited discussions.

I also want to thank Shawn Guzzo for his advice and design feedback, I may have done a few design faux pas were it not for him.

A special thanks to Jason Gregory for his very kind words and taking the time from finishing up his own book to write a bit on mine!

Introduction

Modern games have complex user interface development needs and require skilled game programmers capable of communicating and interfacing with different game systems efficiently. Nowadays, many gameplay and development features are presented as part of the user interface, from mini-games that allow players to complete smaller challenges to unlock and navigate the game universe, to complex statistical models that need to be presented to players, developers or even studio executives.

The user interface is the name given to a broad set of systems that provide players with a bidirectional flow of information. Players need to see the status of the game in real-time, the player’s health, ammo, energy, stamina, and many properties that are changing as often as the game is updated. Visual feedback systems are crucial for players to make decisions about how they navigate the game world, as well as interacting with the game world through a variety of well-known controls and features.

User interface programming is often associated with the development of menus that allow players to configure their gaming experience, while this is true, this is a small part of the big picture. The user interface programmer’s work goes beyond what is seen and perceived by players, user interface developers provide tools to developers to make their development and debugging experience more efficient. We develop systems to show developers a view of the game universe players don’t get to see, hidden enemies, secret rooms, an NPC’s vision cone, the AI’s goal tree, and many more properties that developers need to visualize to understand at a glance what the code is doing underneath the hood.

Whether a dedicated user interface programmer or a game programmer that finds himself doing user interface work, the goal of this book is to provide you with the core knowledge needed to develop user interfaces efficiently and to provide development teams with useful tools that will make their daily work more efficient.

1 Mathematics for Games

The most important skill you can develop if you want to develop user interfaces for games or other media, is mathematics. The range of mathematics used in user interfaces is incredibly broad in 2D space alone, once you take your user interface into your game’s 3D space the amount of mathematics you will encounter grows significantly. A clear understanding of the math means less time spent trying to solve a complex problem through a brute force or a trial and error approach and more time creating interesting and efficient features. One of the most important trait you gain from investing time in understanding the math, is the intuition you develop that often leads you to find and apply reusable mathematical patterns, oftentimes a complex set of if/else conditions can be reduced to a single mathematical formula.

1.1 Notation

In this book and the example code we will use a left-handed coordinate system, with the viewer is looking down the Z axis. We will generally refer to the Z axis as the forward axis, the positive Y axis as the up axis, and the positive X axis as the right.

Figure 1- Left and right handed coordinate systems

Vectors will be presented in column-major form,

Matrices will be also in column major form

The following table lists the conventions used in this book.

1.2 Linear Algebra

When it comes to game development, a powerful tool at our disposal is a good understanding of a few concepts in linear algebra. We will discuss linear algebra in a way in which is applicable to game development, but in particular with an emphasis towards user interfaces.

1.2.1 Vector s

For computer graphics, we are interested in a vector as a geometrical object that contains both a direction and a magnitude (length). Vectors are of great importance in games, you can imagine that every object in every game, whether 2D or 3D has an arrow that starts from its center and points in the direction it is traveling.  This arrow, is a vector, and the length of this vector means how fast this object is traveling. This is perhaps the easiest way to picture what a vector is, vectors and the mathematical operations we can perform on them are useful in a very large number of situations.

Figure 2- A vector denoted by the letter v

Vectors have a number of mathematical operations that can be performed on them, here we will discuss the most common uses and the meaning behind them.

An essential part of any game engine is a comprehensive vector class that offers all the facilities and often used functions in an intuitive way. The goal of this book is to present concepts in a straightforward manner and avoid adding complexity by presenting highly optimized code. We will use different classes vector2, vector3 and vector4 to explicitly state the kind of vector a problem requires, and we will forego some optimizations (such as parallelizing vector hardware) because optimized interfaces are typically more verbose and would distract us from our goal.

Note: The Direct3D example code provides conversion operators between the vector classes and the optimized XMVECTOR, it foregoes the optimization in favor of a more readable and straightforward interface, at the same time this compatibility allows optimized operations to be used in high performance situations and the vector classes used for local storage.

1.2.1.1  Vector Addition

Vector addition is the straightforward operation of adding all the components of one vector to the components of another.

Geometrically, the result can be visualized by placing the second vector at the end point of the first, then drawing a vector between the starting point of the first vector to the ending point of the second.

Figure 3 - Vector addition

The code to add two vectors together is equally straightforward, it is useful to implement the addition operators in the vector class.

1.2.1.2 Vector Subtraction

Just as addition, subtraction is the straightforward operation of subtracting each of the vector’s components.

Vector subtraction is an operation we will find often when working in games, it allows us to calculate a vector between the end points of two other vectors.

Figure 4 - Vector subtraction

This means that if we have two vectors u, v in space representing two points in space, by performing a vector subtraction of u - v we get the vector between these two points.  From this vector we can determine the direction and we are also able to calculate the distance between the two points. One situation in which we may want to do this is to calculate the direction we want to aim the camera in order to look at some point in space.  By performing vector subtraction from the point of interest and the camera’s position, we get the direction vector to which we should orient the camera.

1.2.1.3 Vector Length

A vector represents both a direction and a magnitude, often we need to measure the length (magnitude) of