After my many [hard-learnt-lessons](/projects/mikes-mirror-overview/) building a Smart Mirror the past few months, I decided it might be a good idea to share some of those lessons with others in the community.



# The Talk

At the start of the year I gave [a different talk](talks/a-game-developer-learns-machine-leaning-then-talks-about-it/) at the local Machine Learning Meetup group about my very basic explorations of Machine Learning in game development and it went well and the audience was receptive to non ML-heavy talks so I thought it would be a good place to talk about my mirror project although it wasnt strictly Machine Learning focused, it did involve using some off-the-shelf face recognition and audio recognition ML tools.

I produced a slide deck from my previous condensed blogposts on the topic:

1. [Mikes Mirror - Overview](projects/mikes-mirror-overview/)

2. [Mikes Mirror - Hardware](projects/mikes-mirror-hardware/)

3. [Mikes Mirror - Software](/projects/mikes-mirror-software/)

If you are interested the [slides are available](https://docs.google.com/presentation/d/1nD3MhRzxJPYn63N_mdyxkSFqVFoFneQpcYXBlVICJEE/edit?usp=sharing) and the published show is here:

<iframe src="https://docs.google.com/presentation/d/e/2PACX-1vQ1P3FygWOUhfUXsfnzCyJQoeyjZbVx-Oynq4vPBSVRZgsXp6OPJm2OPNH6o-vK87fkXZrPPEOwKETy/embed?start=false&loop=false&delayms=3000" frameborder="0" width="960" height="569" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe>

The talk went really well. Everyone had a bit of a laugh and I think learnt a little bit about what it takes to build a small IoT device, so im really glad I did it.

For the past 3 years (almost) I have been working as a software developer at TapSlots in Perth Australia. Its been a wild ride of learning and development but over the Christmas break I decided that its about time that I tried something new.



For some time now I have been very interested in Machine Learning and AI. I have been thinking about this ever since reading my first sci-book Isaac Asimov. I, Robot. At 18 I chose not to pursue a university degree in Robotics and AI and instead took my path down a games development route instead.

General purpose AI has always seemed an elusive technology always just outside the grasp of us hairless monkeys. Like fusion power, or a duke-nukem sequel it seems like we are continually just out of reach of the singularity. Recent advancements in deep machine learning however have made it seem that maybe, perhaps maybe we might be getting close.

CNNs, LSTMs, GANs and other ML acronyms de jour are making incredible strides in very specific problems. These arent solutions to the General Purpose AI problem but with them comes incredible interest and funding.

From an entrepreneurial perspective, this sudden surge in AI advancements has brought incredible opportunities for disruption of existing businesses. When combined by cloud powered IoT I dont think there is a single industry that wont be transformed over the next decade or two.

So when I was trying to decide were I should focus my most valuable asset, time, next, I concluded that it was obvious that I should be pointing myself in the direction of AI.

Fortuitously around the same time I was approached to join a new startup in Perth that was using Deep Learning to solve some very interesting computer vision problems for the retail industry. After thinking about it some more I decided that this was an exciting opportunity and aligned well with aspirations and beliefs.

So as of this week I am now the Head of Engineering at GenVis in Perth Australia. Im really excited for the potential of this company and I hope that I can do what I do best, explore and advance new technologies, to bring success in my new role.

To round off my current Machine Learning adventures I gave a talk at a local Machine Learning group.



# Talking about learning

For a while now I have been attending the [Perth Machine Learning Meetup Group (PMLG)](https://www.meetup.com/Perth-Machine-Learning-Group/) and really enjoying the talks by the various incredibly talented data-science and machine learning experts that attend that group. So when they asked for more talks from community members it was with much hesitation that I put my hand up to talk about my minor fumblings in the world of machine learning in game development.

The talk basically went over my posts:

1. [A Game Developer Learns Machine Learning - Getting Started](/machine-learning/a-game-developer-learns-machine-learning-getting-started/)

2. [A Game Developer Learns Machine Learning - A Little Deeper](/machine-learning/a-game-developer-learns-machine-learning-a-little-deeper/)

3. [A Game Developer Learns Machine Learning - Mr Nibbles Basics](/machine-learning/a-game-developer-learns-machine-learning-mr-nibbles-basics/)

The talk went well despite a few internet issues. Most people hadnt done any Unity development before so even thought the ML aspect was probably a little basic for them, at least they will have learnt a little bit about how Unity works :)

I got some great feedback at the end on how I could potentially improve the model's learning through different application of rewards, I took notes and will definately be applying those suggestions next time I do some work on the project.

In my [last post](/machine-learning/a-game-developer-learns-machine-learning-a-little-deeper/) we dug a little deeper into machine learning by starting from scratch and training an agent on a vastly simplified version of mr nibbles. In this post we will explore what happens when we step it up a notch and attempt to train on a significantly more complex version of the game.



# The Series

If you are new to the series I recommend you checkout the previous posts first, here's the whole series:

0. [A Game Developer Learns Machine Learning - Intent](/machine-learning/a-game-developer-learns-machine-learning-intent/)
1. [A Game Developer Learns Machine Learning - Getting Started](/machine-learning/a-game-developer-learns-machine-learning-getting-started/)
1. [A Game Developer Learns Machine Learning - A Little Deeper](/machine-learning/a-game-developer-learns-machine-learning-a-little-deeper/)
1. A Game Developer Learns Machine Learning - Mr Nibbles Basics

# Code

If you want to see the code for this post then checkout this tag: https://github.com/mikecann/MrNibblesML/tree/MrNibblesBasicsBlogPost

# Mr Nibbles Forever vs Mr Nibbles

My original [Intent](/machine-learning/a-game-developer-learns-machine-learning-intent/) was to teach an ML agent to learn my game [Mr Nibbles Forever](http://epicshrimp.com/app/mrnibblesforever/). Based on my experience up to this point I think what might be easier is to begin with trying to train an agent to learn a game similar to "[Mr Nibbles](http://mr-nibbles.com/)" instead of "[Mr Nibbles Forever](http://epicshrimp.com/app/mrnibblesforever/)".

<iframe width="853" height="480" src="https://www.youtube.com/embed/lyAf7VVLdKg" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

Rather than an endless procedurally generated game like Mr Nibbles Forever, Mr Nibbles is a level-based platform-puzzler.

I suspect that Mr Nibbles will be easyier to learn as it's significantly less complex than its later variant, Mr Nibbles Forever.

# The Environment

Before I can even think about training my agent, I need an environment. Mr Nibbles was originally made many years ago in HaXe and thus I will need a way to build levels in Unity.

Unity recently launched a [set of tools](https://docs.unity3d.com/Manual/Tilemap-Painting.html) for generating tile-based games.

So I set to work using those tools to construct some levels:

<iframe width="853" height="480" src="https://www.youtube.com/embed/99TRpAmZFYM" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

I borrowed some code from Unity's [tutorial on platform games](https://unity3d.com/learn/tutorials/topics/2d-game-creation/creating-basic-platformer-game) to build a "Platform Controller" for Mr Nibbles. This is the thing that actually controls the how mr nibbles moves given an input.

I added a little of flourish in the way of big tubes to represent the entrance and exit points for the level. I even wrote some code that animated Mr Nibbles through the tubes but I found that it caused issues in the training so I decided to drop that aspect for now, perhaps ill add it again later.

The end result is that you are able to play a number of levels as mr nibbles. When you reach the exit another level is randomly selected and Mr Nibbles is spawned into that.

<iframe width="853" height="480" src="https://www.youtube.com/embed/IHdPzaLRY98" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

# The Agent

With the levels built its time to code up the agent.

## State

What information do we need to tell the network about the state of the world that will allow it to successfully learn how to navigate it?

Well we obviously need to supply the position of Mr Nibbles and the position of the exit point.

```csharp
public override List<float> CollectState()
{
    var state = new List<float>();

    state.Add(_exitPoint.transform.position.x);
    state.Add(_exitPoint.transform.position.y);

    state.Add(_platformController.transform.position.x);
    state.Add(_platformController.transform.position.y);

    return state;
}
```

We should probably also supply a variable to help with learning the potential set of actions depending if the player is in the air or not.

```csharp
public override List<float> CollectState()
{
    var state = new List<float>();

    ...

    state.Add(_player.IsGrounded ? 1 : 0);

    return state;
}
```

Im not sure if this is really needed but I thought it cant hurt.

Now for the hard part. How do I represent the level? I cant really pass in the collision bounds for each tile as that would be too complicated to represent as floats.

Each level is constructed as tiles, so everything sits on a uniform grid. So I should only need to pass in the centre point of each tile in the level along with its type and hopefully that should be enough for the agent from.

[![](./tile-xy-pos.jpg)](./tile-xy-pos.jpg)

The next issue is; how do I deal with different level sizes? [This GitHub issue](https://github.com/Unity-Technologies/ml-agents/issues/139) mentions that the state size needs to be fixed (I believe this is to do with the way TensorFlow works under the hood).

This restriction is going to make things a little tricky because our levels are different sizes with a different number of tiles, so keeping the state size the same for each level may be a problem (more on potential solutions to that later).

For now my "just get it working" solution is just to make sure the levels are quite small then grab all the tiles in a certain bounds. If the tile is empty I set its type to "0" if its a solid tile its "1" and if its a spider its "2".

[![](./tile-types-rendered.jpg)](./tile-types-rendered.jpg)

When the level changes I grab the tiles from the tilemaps:

```csharp
public BoundsInt tileBoundsToIncludeInState = new BoundsInt(-25, -1, 0, 40, 1, 1);

public override void AgentReset()
{
    ...
    _tiles = _game.CurrentLevel.GetComponentInChildren<TilesController>().GetTiles(tileBoundsToIncludeInState);
    ...
}
```

Then feed them into the state list:

```csharp
public override List<float> CollectState()
{
    ...

    foreach (var tile in _tiles)
    {
        state.Add(tile.position.x);
        state.Add(tile.position.y);
        state.Add(tile.type);
    }

    ...
}
```

## Actions

Now we have our state, we need to allow the agent to take and action and receive a reward or punishment for that action.

```csharp
public override void AgentStep(float[] actions)
{
    PerformActions(actions);
    UpdateRewards();
}
```

Digging into `PerformActions` first:

```csharp
public const int None = 0;
public const int MoveLeft = 1;
public const int MoveRight = 2;
public const int Jump = 3;

...

private void PerformActions(float[] actions)
{
    var nothing = (int) actions[None] == 1;
    var moveLeft = (int) actions[MoveLeft] == 1;
    var moveRight = (int) actions[MoveRight] == 1;
    var jump = (int) actions[Jump] == 1;

    var isJumping = false;
    var hozMove = 0f;

    if (moveLeft)
        hozMove = -1f;
    if (moveRight)
        hozMove = 1f;
    if (jump)
        isJumping = true;

    _platformController.Tick(hozMove, isJumping);
}
```

I handle the condition where the player can perform none, one or all of the actions simultaneously. I then pass those actions to the platform controller component which updates the physics of mr nibbles.

Now we need to handle how to reward the agent for its actions:

```csharp
private void UpdateRewards()
{
    if (_exitPoint.IsTriggered)
    {
        Wins++;
        reward = 10;
        done = true;
    }
    else if (_spiders.IsTriggered)
    {
        Deaths++;
        reward = -10;
        done = true;
    }
    else
    {
        reward = -0.01f;
    }
}
```

The agent is rewarded for reaching the exit, punished for touching the spiders and given a small punishment for each moment that goes by that it doesnt reach the exit.

Im not sure if these are the correct values, I played around with a few different values but im not sure if these are correct or not.

# Defining Brain and Academy

[As with my last post](/machine-learning/a-game-developer-learns-machine-learning-a-little-deeper/) I am going to leave the academy empty for now, I dont need its features as I dont have multiple agents in my scene.

The brain is setup like so:

[![](./brain-setup.png)](./brain-setup.png)

We have defined the number of states: tiles (x,y,type) + agent state (x,y,isOnGround) + exit point (x,y) = 1505

The number of actions: jump, left, right, nothing (stand still).

The action space is set to continuous because we want to allow the agent to perform multiple actions at once, jump and move right for example.

The state space is continuous because we arent dealing with integer positions for things in the level rather we are using floats.

# Training

With the environment constructed lets move onto training. First I need to get the hyperparameters right for the PPO algorithm. I re-read [the unity best practices doc](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/best-practices-ppo.md) and took a guess at the following hyperparams:

```python
### General parameters
max_steps = 5e5 # Set maximum number of steps to run environment.
run_path = "mrnibbles/run14-differentrewards" # The sub-directory name for model and summary statistics
load_model = False # Whether to load a saved model.
train_model = True # Whether to train the model.
summary_freq = 10000 # Frequency at which to save training statistics.
save_freq = 50000 # Frequency at which to save model.
env_name = "mrnibbles" # Name of the training environment file.
curriculum_file = None

### Algorithm-specific parameters for tuning
gamma = 0.99 # Reward discount rate.
lambd = 0.95 # Lambda parameter for GAE.
time_horizon = 1500 # How many steps to collect per agent before adding to buffer.
beta = 1e-3 # Strength of entropy regularization
num_epoch = 6 # Number of gradient descent steps per batch of experiences.
num_layers = 2 # Number of hidden layers between state/observation encoding and value/policy layers.
epsilon = 0.2 # Acceptable threshold around ratio of old and new policy probabilities.
buffer_size = 10000 # How large the experience buffer should be before gradient descent.
learning_rate = 3e-4 # Model learning rate.
hidden_units = 64 # Number of units in hidden layer.
batch_size = 2500 # How many experiences per gradient descent update step.
normalize = False
```

I kicked off training and fired up TensorBoard. After a few different attempts I felt like I had something that was training so I let it run for an hour or so:

[![](./tensorboard.png)](./tensorboard.png)

It looks okay, we can see that the cumulative reward increases sharply and levels off and the episode length decreases as the agent learns to play the level.

When we try our model out in the game however we see this:

<iframe width="853" height="480" src="https://www.youtube.com/embed/S21Rkob0XfM" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

On the positive side, it looks like he is attempting to move towards the exit point when the exit is to the right of the spawn point. On the negative side he doesnt ever seem to want to try going left and also seems to get stuck for a long time at a small hurdle.

I think whats going on is the levels where the exit point is to the left of the spawn point have spiders, thus they are significantly more difficult for the agent to solve than the levels where the exit point is to the right of the spawn point. As a result the agent has learnt that probably it should always just go right and is unable to learn that it can jump over the spiders.

It looks like the agent is having trouble with jumping in general..

# Future Work

I tried a few smaller tweaks but none seemed to have a significant effect. I think what I need to to change my training strategy.

## Curriculum Training

While browsing through the Unity docs I came across the concept of [curriculum training](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/curriculum.md). As I understand it; you incrementally train your agent with progressively more difficult scenarios thus accelerating the learning by focusing the agent on learning just one aspect of your game at a time. This reminds me of how humans learn in school, we aren't immediately thrown into quantum mechanics, instead we start with basic mathematics and work our way up from there.

I think my next set of experiments will have to take advantage of curriculum learning by slowly introducing level elements until the agent is able to successfully proceed.

## Tile Radius State

One current limitation of my current setup is that I encode the state of the entire level. This means I am currently limited to small levels of fixed size.

A solution (suggested by a member of [my local Machine Learning meetup group](https://perthmlgroup.wordpress.com/)) is to only provide the state of tiles that lie in a radius around Mr Nibbles. As Mr Nibbles moves so to does the nearest surrounding tiles. This is analogous to using a camera as my input state but unlike camera state the agent should be able to learn [temporal information](https://github.com/Unity-Technologies/ml-agents/issues/183).

## Reward Tweaking

I think I may need to work on how I reward and punish a little bit more. Perhaps I should have a timeout if it takes the agent too long to solve a level, maybe that would prevent it from learning that on certain levels its best just to suicide rather than try to solve the level.

## Velocity

Another potential issue is was unable to learn to jump correctly was because it was missing information about its velocity. I was hoping it could infer the velocity based on the previous positional state of mr nibbles but perhaps it would help if I manually supplied that information as two more state parameters.

# Conclusion

Well I cant really call this post a success, but I certainty learnt a lot and that's the ultimate goal. I have lots of ideas for how to proceed from here so subscribe to stay tuned.

Until next time!

P.S. If you want to see the code for this post then checkout this tag: https://github.com/mikecann/MrNibblesML/tree/MrNibblesBasicsBlogPost

I mentioned [in my last post](/machine-learning/a-game-developer-learns-machine-learning-intent/) that the intent is to write an AI to play [Mr Nibbles Forever](http://epicshrimp.com/app/mrnibblesforever/) better an any human. Before we can get anywhere near that goal however we need to get the very basics working first.



# The Series

If you are new to the series I recommend you checkout the previous posts first, here's the whole series:

0. [A Game Developer Learns Machine Learning - Intent](/machine-learning/a-game-developer-learns-machine-learning-intent/)
1. A Game Developer Learns Machine Learning - Getting Started
1. [A Game Developer Learns Machine Learning - A Little Deeper](/machine-learning/a-game-developer-learns-machine-learning-a-little-deeper)
1. [A Game Developer Learns Machine Learning - Mr Nibbles Basics](/machine-learning/a-game-developer-learns-machine-learning-mr-nibbles-basics)

# TLDR;

We will use Reinforcement Learning to train an agent to balance a ball.

<iframe width="853" height="480" src="https://www.youtube.com/embed/ZnBfvARKXeo" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

# Unity ML Agents

I find that when tacking something new for the first time, the best thing to do is take something built by others, then hack around with it for a little bit to understand the very basics. So rather than starting from total scratch im going to look for something that already exists.

As luck would have it, Unity have [just released](https://blogs.unity3d.com/2017/09/19/introducing-unity-machine-learning-agents/) a new set of tools to help game developers use machine learing in their games, perfect!

# Getting Setup

First things first, I clone the Unity ML Agents repo https://github.com/Unity-Technologies/ml-agents.

Then I start following their [Getting Started With Balance Ball](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Getting-Started-with-Balance-Ball.md) docs.

The first thing I need to do is get my system setup with the neccessary tools for machine learning. I follow this guide to do so: https://unity3d.college/2017/10/25/machine-learning-in-unity3d-setting-up-the-environment-tensorflow-for-agentml-on-windows-10/

This process takes quite a while, make sure you follow the instructions EXACTLY. At the bottom of that page is a video which is helpful.

<iframe width="853" height="480" src="https://www.youtube.com/embed/qxicgknzUG8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

# Jupyter Notebook

So with my system now setup I return back to the [Balance Ball](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Getting-Started-with-Balance-Ball.md) doc.

The next step is to open Jupyter Notebook which is a popular tool with ML community for training models.

[![](./jupyter-basics.png)](./jupyter-basics.png)

Its actually a pretty aweomse tool. Its sort of like a [REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop) mixed with word document.

You can write code right inside of the doc without firing up and IDE and then execute it and see the results inline.

# Testing the Bridge

Next up is the "Testing Python API" section of the docs.

In Unity the balance ball scene is compiled into an exe, this is whats known as the "environment".

[![](./showing-exe.png)](./showing-exe.png)

Python in the notebook talks to the "brain" in the environment. Its a sort of native bridge between the machine learning API in Python and your "game" / "environment".

[![](./bridge.png)](./bridge.png)

When the training session is started from python, the environment exe is executed which opens as a separate window on your computer. This surprised me as I was expecting it to be hidden background process of some sort.

The environment window that opens up is tiny, presumably so your GFX card doesn't have to waste cycles on rendering and can instead focus on ML.

[![](./env-window.png)](./env-window.png)

The "brain" in the environment is able to receive commands which are called "actions". When an action runs, it changes the "state" of the environment which is returned back to Python. That process of `State > Action > New State` is repeated over and over again (along with some other magic) and out pops our agent.

[![](./learn-loop.png)](./learn-loop.png)

# Training the Model

Now we know our bridge from Pythong to Unity works okay we can move onto actually training the model.

Folling the "Training with PPO" section of the docs we open "TensorBoard" which gives us stats on the the training process as it runs.

[![](./tensorboard1.png)](./tensorboard1.png)

Im not really sure what all those stats mean but thats okay at this stage, we are just tinkering!

The main thing is my model is busy training itself as evidenced by the changing graphs and my pegged CPU 😜

[![](./pegged-cpu.png)](./pegged-cpu.png)

# Testing the Model

Once the "cumulative_reward" is above 75 the model is ready to rock. So I stop the training process in Jupyter.

Theres one last step before we can run the model in Unity, we must add support for TensorFlow directly in Unity in the form of the TensorFlowSharp library.

[![](./tf-sharp.png)](./tf-sharp.png)

TensorFlowSharp requires the experimental .Net 4.6 support in Unity. Thats something to note because as my goal is to eventually get this working on Mr Nibbles Forever which is built on the old .Net 3.5

With the library installed we copy over the generated model into the game and huzzah! It works!

[![](./it-works.png)](./it-works.png)

# Adding some Dynamics

Its awesome that it works but its hard to tell if this is actually balancing the ball dynamically or not. I think if we add a dynamism it will make things clear.

Lets add this script to each of the platforms:

```csharp
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class BallNudger : MonoBehaviour
{
    public Color nudgeColor = Color.red;
    public float forceMultiplier = 5f;
    public float colorChangeTime = 0.5f;

    private bool _isNudging = false;
    private Renderer _renderer;
    private Rigidbody _rigidBody;
    private Color _startingColor;
    private float _colorChangeAge;

    void Awake()
    {
        _renderer = GetComponent<Renderer>();
        _rigidBody = GetComponent<Rigidbody>();
        _startingColor = _renderer.material.color;
        _colorChangeAge = colorChangeTime;
    }

    void OnMouseDown()
    {
        _colorChangeAge = 0;
        var ray = Camera.main.ScreenPointToRay(Input.mousePosition);
        RaycastHit hit;
        if (Physics.Raycast(ray, out hit))
            _rigidBody.AddForceAtPosition(ray.direction * forceMultiplier,
                hit.point, ForceMode.Impulse);
    }

    void Update()
    {
        _colorChangeAge += Time.deltaTime;
        _renderer.material.color = _colorChangeAge < colorChangeTime ?
            nudgeColor : _startingColor;
    }

    private Vector3 GetRandomForce()
    {
        return new Vector3(
            Random.Range(-forceMultiplier, forceMultiplier),
            Random.Range(-forceMultiplier, forceMultiplier),
            Random.Range(-forceMultiplier, forceMultiplier));
    }
}
```

Now when we click the ball, it changes color and gets a little nudge. Lets run it again and yes, indeed the AI is able to balance the ball dynamically as the input changes.

<iframe width="853" height="480" src="https://www.youtube.com/embed/ZnBfvARKXeo" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

# Conclusion

Cool! Well that wasn't as hard as I thought it was going to be.

Sure I haven't actually done any of the hard work or really learnt much about the fundamentals yet but at least I have something working, ready for me to hack with.

So stay tuned, subscribe below as next episode im going to dive a little deeper.

Lately I have spent a considerable amount of time thinking about where I want to go in my career and what sort of things do I want to work on going forward.



# The Series

This is a series of multiple parts, if you liked this post then checkout the others:

0. A Game Developer Learns Machine Learning - Intent
1. [A Game Developer Learns Machine Learning - Getting Started](/machine-learning/a-game-developer-learns-machine-learning-getting-started/)
1. [A Game Developer Learns Machine Learning - A Little Deeper](/machine-learning/a-game-developer-learns-machine-learning-a-little-deeper)
1. [A Game Developer Learns Machine Learning - Mr Nibbles Basics](/machine-learning/a-game-developer-learns-machine-learning-mr-nibbles-basics)

# The History

[![](/wp-content/uploads/2007/09/Guess.jpg)](/wp-content/uploads/2007/09/Guess.jpg)

Before you think about where you want to go next sometimes it helps to think about where you have come from.

When I was 7 years old I [coded my first game](http://localhost:4500/about/). I loved the thrill of turning a meer figment of my imagination into something I could see on the screen and delight when shown to others.

At 18 I went to University and studied Computer Games Programming and loved every second of it, far surpassing my performance during high school, eventually graduating with a first degree with honours.

For the next 10+ years I proceeded to work in the games industry first starting in the flash games world then social games before moving to mobile.

Games are not the only thing I have been interested in over the years as this blog [certainty can attest to](/all-tags/). However games are definitely my core competency.

# The New Shiny

So when you are in an industry that moves so fast with so many new technologies being released every day, it can be really hard to know where to spend your time, particularly for someone like me who "just cant stop tinkering with things" ™.

Machine Learning (ML) is one of the biggest technologies to emerge over the past couple of years. I honestly think that ML has the power to transform every single industry from finance, to agriculture to sports science.

I also think that the games is no different from the others and wont be spared from its transformative effects. Some think that the games industry is safe because its a "creative" industry and thus not susceptible to being replaced by a logic driven machine.

Instead I see a future where an AI is capable of generating an entire game based on your preferences such as "I would like a game that plays like Mario but has the same size and scope as World of Warcraft", the AI then generates variants on that for you to play.

Machine learning opens so many potential possibilities and opportunities for someone with the idea and ability to explore them.

So with that in mind I think that it behove me to at least consider exploring ML a little more.

# The Concern

<div style="width:100%;height:0;padding-bottom:56%;position:relative;"><iframe src="https://giphy.com/embed/rdvtXVH7jDjKU" width="100%" height="100%" style="position:absolute" frameBorder="0" class="giphy-embed" allowFullScreen></iframe></div><p>

But will I enjoy it? This is a major concern of mine. From what I have seen thus far, machine learning is very math heavy, invloves a lot of Python (dynamic programming, _shudders_) and isnt particularly visual. All negatives for someone who isnt great at maths, doesnt really like dynamic languages and is a games programmer.

I am also concerned that this I will basically be starting from scratch again. All my games programming knowledge counts for nothing. Will I be able to adapt to this new set of problems, solutions and tools?

So I think only way to know for sure tho is just to try it and see and the best way to try is to tinker! ™

# Learn to Learn

So over the next few months im going to dig deeper into machine learning. Try to understand it how it works and how it applies to game development.

Im going to blog each step of my adventures. This serves two purposes, firstly I find that writing things down really helps me understand things at a deeper level and secondly I hope that if I dont do a terrible job at explaining it will help others to learn too.

Im going to call this series of blog posts <strong>"A Game Developer Learns Machine Learning"</strong>.

# The Series Goal

[![](https://www.mikecann.blog/wp-content/uploads/2016/05/MrNibblesForever_Gif_5.gif)](https://www.mikecann.blog/wp-content/uploads/2016/05/MrNibblesForever_Gif_5.gif)

The goal for the series is I want to be able to write an AI that is able to play my game [Mr Nibbles Forever](http://epicshrimp.com/app/mrnibblesforever/) better than any other human player.

I have no idea if thats an achievable goal but I think its a goal that should give me a solid understanding of the fundamentals of reinforcement learning and give me a target with which to focus on and guide me through this dense but wonderful topic.

So if you are interested in following along, either [add this blog to your favourite RSS reader](/atom.xml) or enter your email below to subscribe and get email updates.

Wow, I cant believe its been one whole year since [the last Govhack](https://www.mikecann.blog/portfolio/projects/govhack-2015-should-i-drive/), the hackathon where groups of people use government data to hack together a project over the course of a weekend.



I had a great time last year on the "[Should I Drive?](https://www.mikecann.blog/portfolio/projects/govhack-2015-should-i-drive)" team. We used WA Main Roads and other data sources to try to answer the question "should I drive to my destination or take some other form of transport?".

This year I set myself a goal. I wanted to do something to do with Machine Learning / AI. Its a field of computing thats really hot right now and im really interested in getting involved and learning more.

So as with [last year](https://www.mikecann.blog/portfolio/projects/govhack-2015-should-i-drive), after a brief welcome presentation by the organizers, competitors were invited to take the microphone and pitch ideas. Off-stage the pitchers then threw together a quick poster with their main ideas.

[![2016-07-29 18.52.00](https://www.mikecann.blog/wp-content/uploads/2016/08/2016-07-29-18.52.00-1024x768.jpg)](https://www.mikecann.blog/wp-content/uploads/2016/08/2016-07-29-18.52.00.jpg)

There were a coupple of interesting ones but the one that really caught my attention was from a guy who wanted to apply Machine Learning to old historic photographs. After a brief discussion with him around his poster I immediately signed up. 20 minutes later we had a team and were heading off upstairs to find a quiet area of the (awesome) [Flux co-working space](https://www.fluxperth.com/) where Govhack was being held this year.

In total we had 8 members, 3 technical and 5 non-technical. That first evening was mostly spent planning who was going to do what and what the priorities were. The three of us technical people sketched out our plan and divvied up the work so that we were all working efficiently.

[![3tech](https://www.mikecann.blog/wp-content/uploads/2016/08/3tech-1024x683.jpg)](https://www.mikecann.blog/wp-content/uploads/2016/08/3tech.jpg)

I took the front-end website and backend node host / API while Dominic took the Python code which would interface with the various data sets and Houraan did the code which would apply the Machine Learning to the images returned from the data sets.

[![theplan](https://www.mikecann.blog/wp-content/uploads/2016/08/theplan-1024x683.jpg)](https://www.mikecann.blog/wp-content/uploads/2016/08/theplan.jpg)

With the plan sorted I knew I had a bunch of work to get done before the non-technical people could get their hands on something to use and experiment with. With that in mind I decided to get up really early (4am) on Saturday morning and start work on the site. Im glad I did as it took me most of the morning to get the shell of an app up and running using ReactJS with Typescript and a NodeJS backend hosted on Heroku (such is the way with modern Javascript).

[![chrome_2016-08-01_12-54-14](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_12-54-14-1024x830.png)](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_12-54-14.png)

When I arrived at Flux later that morning, I demoed my progress and we discussed the scope of the project. The original idea was to produce videos from several photos in a sort of slideshow but after some discussion we decided to narrow the scope so that we were more likely to finish it in time. We decided that if we could just take old photos and apply ML to "Colourise" them then would be a cool way to explore the past using a modern technique.

With the scope of the project resolved, our next task was to come up with a name for it. One of our non-technicals Karl came up with "Colourful Past", we all agreed that it fit the scope and described the project perfectly.

The rest of the day was spent furiously hacking away on various facets of the project.

We setup a [Trello Board](https://trello.com/b/en3HeDLx/colourful-past) to manage tasks and a place to store links and other information about the project.

[![chrome_2016-08-01_13-04-34](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_13-04-34-1024x804.png)](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_13-04-34.png)

We used [Slack](https://slack.com/) for general communications and link sharing when we couldnt just shout across the table.

[![slack_2016-08-01_13-48-55](https://www.mikecann.blog/wp-content/uploads/2016/08/slack_2016-08-01_13-48-55-1024x718.png)](https://www.mikecann.blog/wp-content/uploads/2016/08/slack_2016-08-01_13-48-55.png)

Source code was uploaded to our github org [https://github.com/colourful-past](https://github.com/colourful-past):

[![chrome_2016-08-01_15-40-43](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_15-40-43.png)](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_15-40-43.png)

In general things went really smoothly. The technical side worked really well, we were able to work efficiently independently then combine the results towards the end. On the non-technical side there few a few issues managing tasks, keeping everyone working all the time but in general we were able to work effectively together.

By the end of the first day we had a basic but working product. You could type in a search term such as "Anzac Day", the client would then send an API request to the NodeJS server which would then in parallel call a number of Python scripts to query various datasets, the results were then aggregated and returned to the client.

[![chrome_2016-08-01_14-28-58](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_14-28-58.png)](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_14-28-58.png)

The user could then click a button to "Colourise" the black and white image. This makes another call to NodeJS which calls another Python script which uses a Machine Learning model developed by UC Berkeley and trained on 1.3 million black and white images to generate a coloured version of our historic photograph. The resulting image is stored in S3 and the URL returned to the client.

[![chrome_2016-08-01_14-35-48](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_14-35-48.png)](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_14-35-48.png)
(above image is just a placeholder and was not generated by the AI)

After I left on the Saturday evening, to get some needed sleep, Houraan soldiered on and gave the site some much needed design love. When I woke in the morning the site looked much improved, Houraan had done a phenomenal job.

[![chrome_2016-08-01_14-41-27](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_14-41-27.png)](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_14-41-27.png)

[![chrome_2016-08-01_14-42-59](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_14-42-59.png)](https://www.mikecann.blog/wp-content/uploads/2016/08/chrome_2016-08-01_14-42-59.png)

Sunday was our final day and we spent the first half adding the last bit of polish to the site, such as a really cool subtle gradient effect on the text:

[![2016-08-01_14-48-13](https://www.mikecann.blog/wp-content/uploads/2016/08/2016-08-01_14-48-13.gif)](https://www.mikecann.blog/wp-content/uploads/2016/08/2016-08-01_14-48-13.gif)

Then we concentrated on the presentation and video which the judges will be using later that day. Molly did a great job putting together our video which was uploaded to Youtube at the very last minute:

<iframe width="853" height="480" src="https://www.youtube.com/embed/kuNnUWMXwXs" frameborder="0" allowfullscreen></iframe>

The presentation worked a little different from last year. Instead of everyone getting up on a stage and doing a slideshow in front of the judges, the judges came round to each team's desk where we did a demo and a short talk before being asked a number of questions. Houraan, Karoline and Bruce nailed it for us while the rest of us watched and gave moral support.

[![2016-07-31 15.33.01](https://www.mikecann.blog/wp-content/uploads/2016/08/2016-07-31-15.33.01-1024x862.jpg)](https://www.mikecann.blog/wp-content/uploads/2016/08/2016-07-31-15.33.01.jpg)
(We had a little spare time so we thought we would play on the "Colourful" aspect of our product a little by blowing up some balloons, producing a poster and dressing in bright colours)

All in all it went really well and im very happy with the result. The judges seemed to think so too as we came away with a 1st prize in the "West Australian Community Prize" category.

[![2016-07-31 19.33.43](https://www.mikecann.blog/wp-content/uploads/2016/08/2016-07-31-19.33.43-1024x768.jpg)](https://www.mikecann.blog/wp-content/uploads/2016/08/2016-07-31-19.33.43.jpg)

Well thats just about it. If you want to have a look at what we produced you can try it out at [https://colourfulpast.org/](https://colourfulpast.org/). We dont know how long we are going to be able to keep the expensive AWS GPU instances going for so if you are viewing this post some time from now it might not work for you.

I just want to say a massive thanks to all my team-mates for making it an awesome weekend of fun hacking, thanks guys, I hope you see you all next year!

[![IMG_0389](https://www.mikecann.blog/wp-content/uploads/2016/08/IMG_0389-300x200.jpg)](https://www.mikecann.blog/wp-content/uploads/2016/08/IMG_0389.jpg)[![IMG_0455](https://www.mikecann.blog/wp-content/uploads/2016/08/IMG_0455-300x200.jpg)](https://www.mikecann.blog/wp-content/uploads/2016/08/IMG_0455.jpg)[![IMG_0436](https://www.mikecann.blog/wp-content/uploads/2016/08/IMG_0436-300x200.jpg)](https://www.mikecann.blog/wp-content/uploads/2016/08/IMG_0436.jpg)[![IMG_0395](https://www.mikecann.blog/wp-content/uploads/2016/08/IMG_0395-300x200.jpg)](https://www.mikecann.blog/wp-content/uploads/2016/08/IMG_0395.jpg)

(P.S. Big thanks to Karl for taking all the pictures!)