RETURN TO VARUNRAMESH.NET

Editing Gameplay Videos without Re-encoding using FFmpeg

Tuesday, December 26th 2017


I recently worked on a Lego Island 2 let’s play with my brother. It was recorded using Dxtory with the x264vfw codec, meaning that the saved recordings are H.264 streams in an AVI container1. Our recordings are 1920x1080 at 60fps. Audio commentary was recorded separately in Audacity.

When it came time to edit the videos, I fired up Adobe Premiere2, but quickly ran into a problem. Rendering the 1080p/60fps videos was taking upwards of an hour. Futhermore the output videos had significantly lower quality due to the re-encoding. I knew that after YouTube transcoded the videos for it’s internal format, the final output would look even worse. To fix this, I did some experimenting with FFmpeg, a command-line based video processor, and found a worflow for editing our let’s play without re-encoding.

These tips require the command-line, but if you can get past that barrier, you’ll be able to edit your videos without long rendering times and quality downgrades. Also, you can do this for free without purchasing any software!

Trimming Footage

You can use FFmpeg to trim footage off of the beginning and end of your video. Below is an example of trimming a 20-second clip from the 100 second mark 3.

ffmpeg -ss 100 -i input.avi -t 20 -c copy output.avi

If you run this, you might find that your video isn’t exactly 20 seconds, but a little bit longer. This is because this method trims the video starting from the nearest keyframe to the provided timestamp. Unfortunately, there’s no way to get around this without re-encoding.

Concatenating Footage

Now let’s say you have two separate gameplay recordings that you want to concatenate together. FFmpeg lets you do this without having to re-encode. The interface for this is a bit strange - you have to create a file with file '{VIDEO_FILE_NAME}' on each line. Here’s a snippet for cutting two videos out of a source file and concatenating them together:

ffmpeg -ss 100 -i input.avi -t 20 -c copy output_1.avi
ffmpeg -ss 200 -i input.avi -t 20 -c copy output_2.avi
echo "file 'output_1.avi'\nfile 'output_2.avi'" > concat_list.txt
ffmpeg -f concat -safe 0 -i concat_list.txt -c copy concatenated.avi

Adding in Audio Commentary

This is the snippet that I use to mix in audio commentary with game audio, assuming that the audio commentary is already synced with the video. It’s a bit complicated, as it uses FFmpeg’s filtergraph functionality. Note that the audio is encoded using the mp3 codec. I keep my audio in FLAC form up till this point, so this is the first point where the audio is encoded.

ffmpeg -i input.avi -i commentary.flac \
    -filter_complex "[0:a]volume=0.5[volumeadj];[volumeadj]aformat=sample_fmts=s16:channel_layouts=stereo[volumeadj_fmt];[volumeadj_fmt][1:a]amerge=inputs=2[merged];[merged]volume=2.0[aout]" \
    -map 0:v -map "[aout]" \
    -c:v copy \
    -c:a libmp3lame -ac 2 \
    output.avi

However, this assumes that the audio is already synced. To sync it, you have two options:

  1. Pad silence at the beginning of your recording in Audacity.
  2. Concatenate silence to the beginning of the audio in FFmpeg, which is shown in the snippet below.
ffmpeg \
  -t 10 -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=44100 \
  -i commentary.flac \
  -filter_complex "[0:a][1:a]concat=n=2:v=0:a=1[out]" \
  -map "[out]" \
  -c:a flac -ac 2 \
  output.flac

To find sync points, what I’ve done is move the cursor up and down in the menu, while saying the words “up” and “down.” This gives me a point in both the commentary and game-play recording to sync up 4.

Putting it All Together

You shouldn’t use this process for videos with a lot of edits as it’s much easier to use an NLE such as Premiere or Vegas 5. However, if you’re creative about how you split-up and sync your videos, I think this approach is worth it. The output videos are the same quality as your raw recordings, and the encoding process is often faster than real-time. Check out our let’s play below!

More Complicated Editing

Here’s an example of a more complicated edit involving multiple overlays and zoom levels that show up at different times. The entire edit, including trimming, audio syncing, and audio merging is done in one FFmpeg command. Since overlays are applied to the video, it must be re-encoded.

ffmpeg \
    -framerate 60 -loop 1 -i "zoomed-out-frame.png" \
    -i "gameplay-recording.ts" \
    -framerate 60 -loop 1 -i ".zoomed-in-frame.png" \
    -i "commentary.flac" \
    -t 4.11 -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=44100 \
    -filter_complex "[1]scale=1024:640:flags=neighbor[scaled];[0][1]overlay=25:144:shortest=1[zoomedout];[2][scaled]overlay=50:38:shortest=1[zoomedin];[zoomedout][zoomedin]overlay=shortest=1:enable='gte(t,105)'[out];[3:a]atrim=19.131[trimmedaudio];[4][trimmedaudio]concat=n=2:v=0:a=1[adjustedaudio];[1:a]volume=0.35[volumeadj];[volumeadj][adjustedaudio]amerge=inputs=2[aout]" \
    -map "[out]" -map "[aout]" \
    -ss 00:00:21 -to 00:25:07 \
    -c:v libx264 -crf 18 \
    -c:a libmp3lame -ac 2 \
    -y output.mp4

And the resulting video…

For this kind of video, I highly recommend using an NLE!


  1. The CRF is set to 23 (the default). I also had to increase the encoding speed, at the expense of having larger files. 

  2. I’m using CS5.5 

  3. More info on trimming videos with FFmpeg

  4. Once I forgot to do this and had to sync up by trying to match button press sounds with in-game actions! 

  5. The reality is that, right now, there aren’t any good open-source NLEs so you’ll have to open your wallet for that kind of editing. The most promising one that I’ve looked at is OpenShot, and it just got an update that’s supposed to improve stability. 

View Comments

How I Structure GameObjects - Components and Mixin-based Inheritance

Thursday, November 2nd 2017


Component-based GameObjects are similar to the concept of mixins/traits in some Object-Oriented langauges. In this article, I’ll discuss the relative advantages and disadvantages of both patterns, then describe how they can be unified in a way that takes the best of both patterns.

A Quick Review of Component-based Game Objects

Many of you who have used Unity or other game engines are familiar with the component-based model of GameObjects. You can skip ahead to the next section if you are. If you don’t know what it is or want a review, then keep reading!

Many people write their games using traditional single-inheritance object heirarchies, but it turns out single-inheritance can rarely describe all of the ways one might want to reuse code in a game.

Suppose you have a simple game with a player and some enemies. All players have collision detection, meaning that they cannot pass through walls. Only some enemies have collision detection, while others are ghosts that can pass through walls. You would like to reuse the collision detection code across the entities that use it, as well as share code across enemy entities, but is this even possible using single inheritance? Take a look at the two attempts at a hierarchy below. Grey blocks represent abstract classes.

If you split GameObjects into Players and Enemies, then you will end up with Player and CollidingEnemy duplicating collision code. If you split GameObjects into Colliding and non-colliding entities, then CollidingEnemy and GhostEnemy will end up sharing enemy-related code.

The solution is to simply create a CollisionComponent class. Player and CollidingEnemy will have CollisionComponent objects as members, and can call into that object to perform operations related to collision. 1

This model is explored in more detail (with C++ code) in Game Programming Patterns by Bob Nystrom.

Problems with Component-based Objects

The component-based GameObject pattern has some problems. Some architectures like Unity are so pure that GameObjects cannot be subclassed at all, and will always be an empty container for components. This is not ideal, as you end up having many one-off components that are only used by a single object.

If you allow both models - subclassing GameObjects and adding in Components, you can create a development flow whereby you first write behavior in a GameObject subclass. Then, when you find yourself writing similar code across multiple game objects, you can refactor it into a component, which you can then add in to your GameObject and use.

Another issue with Components is that they are members, so you have to call this.component.DoSomething() anytime any interesting action should occur. Furthermore, calling from one component into another component on the same object requires something like this.gameObject.otherComponent.DoSomethingElse().

Mixin-based Game Objects

I first encountered Mixins as an alternative to components in this blog post. In this case, you write your reusable components as mixins, which are then included into a class 2. Here’s a concrete example taken from that post:

Dust = class('Dust', Entity)

Dust:include(PhysicsRectangle)
Dust:include(Timer)
Dust:include(Fader)
Dust:include(Visual)

function Dust:init(world, x, y, settings)
    Entity.init(self, world, x, y, settings)
    self:physicsRectangleInit(self.world.world, x, y, 'dynamic', 16, 16)
    self:timerInit()
    self:faderInit(self.fade_in_time, 0, 160)
    self:visualInit(dust, Vector(0, 0))

    // ...
end

function Dust:draw()
    if debug_draw then self:physicsRectangleDraw() end
    self:faderDraw()
    self:visualDraw()
end

The mixins, such as PhysicsRectangle and Timer add functions onto the object that can then be called using self:mixinFunc(...) as if they were defined in the class. More details can be found in the blog post. This solves the problems mentioned above with the component-based model, as you can still use inheritance, this.component.DoSomething() becomes this.DoSomething(), and this.gameObject.otherComponent.DoSomethingElse() becomes this.DoSomethingElse(). However, we’ve introduced some new problems.

Unifying Component-based Objects and Mixins

It turns out you can have your cake and eat it too - effectively getting the best of both worlds. Let’s start with a component-based implementation of GameObjects.

function GameObject:addComponent(comp)
  comp:setGameObject(self) -- Set the owning GameObject of this component.
  table.insert(self.components, comp) -- Add to components table.
  return comp -- Return the component for chaining.
end

-- Draw and update simply call out to the components.
function GameObject:draw()
  for _, comp in ipairs(self.components) do comp:draw() end
end
function GameObject:update(dt)
  for _, comp in ipairs(self.components) do comp:update(dt) end
end

Now take a look at this GameObject:includeComponent function.

--[[ 'Installs' a component into the current object by delegating all methods
to it. Returns the component for compositional purposes. ]]--
function GameObject:includeComponent(comp)
  self:addComponent(comp) -- Add the component, so that it's draw, update, etc. functions will be called.
  for key, value in pairs(comp.class.__instanceDict) do
    if type(value) == "function" and not string.starts(key, "__") and
      key ~= "draw" and key ~= "update" and key ~= "initialize" then
        self[key] = function(_, ...)
          return comp[key](comp, ...)
        end
    end
  end
  return comp -- Return the component.
end

What this does is two-fold:

Now, for every component, we can either “add” it, or “install” it. “Installed” components can be accessed using functions directly on the current object itself, as if it was a mixin. But unlike a mixin, we can still deal with the case where we need two components of the same type.

Heres some code that constructs a guard out of a few components.

ShotgunGuard = class('ShotgunGuard', Entity)

function ShotgunGuard:initialize(layer, pos)
  Entity.initialize(self, 'enemy', layer, pos)

  self.image = self:addComponent(ImageComponent(loader.Image.guard))
  self.shotgun = self:addComponent(ImageComponent(loader.Image.shotgun))

  self.unit = self:includeComponent(UnitComponent(8))
  self.healthBar = self:includeComponent(HealthBarComponent(vector(0, -75)))
end

Recap

So let’s recap what we have…

Right now, this is how I structure my GameObjects, and I feel like it’s a strong way going forward for using components but also limiting the verbosity of calls into those components.


  1. This can be seen as a specific instance of composition over inheritance. 

  2. Mixins and traits are often called “horizontal inheritance.” 

View Comments

Lua Gotchas

Friday, October 20th 2017


Lua is an awesome language, but it has some unusual design choices that can cause endless frustration to beginners (and experts). This post goes over these gotchas 1.

Variables are Global Unless Specified with local

In Lua, variables are globally scoped by default. This has lead to many problems for me, since a temporary variable used by a function can be overwritten by anything called by that function. Here is a somewhat contrived example, where two functions both use the variable name temp as an intermediate value, thereby overwriting each other. Running this snippet and entering 3, and 4 produces the result 8, when it should return 7. Try this snippet on repl.it!

function readNumber()
  print("Type a number:")
  temp = io.read("*number")
  return readAnotherNumber() + temp
end

function readAnotherNumber()
  print("Type another number:")
  temp = io.read("*number")
  return temp
end

We can fix this by adding the local modifier to the temp assignment, as follows:

function readNumber()
  print("Type a number:")
  local temp = io.read("*number")
  return readAnotherNumber() + temp
end

function readAnotherNumber()
  print("Type another number:")
  local temp = io.read("*number")
  return temp
end

Here’s something from one of my projects - its a backtracing search. Can you spot the bug?

function best_next_action(actions_so_far)
  if #actions_so_far == SEARCH_DEPTH then
    return nil, eval_actions(actions_so_far)
  end

  best_action, best_score = nil, -math.huge
  for next_action=-1, 1, 2/(STEERING_BINS - 1) do
    table.insert(actions_so_far, next_action)
    _, score = best_next_action(actions_so_far)
    if score > best_score then
      best_score = score
      best_action = next_action
    end
    table.remove(actions_so_far)
  end

  return best_action, best_score
end

I forgot to make best_action and best_score local, thereby making recursive calls destroy the caller’s value. This bug took hours to find. score should also be made local, though in this situation it’s not a problem.

I recommend you do yourself a favor and get in the habit of making every variable a local by default.

Functions are Global Unless Specified with local

Similar to variables, function in Lua are global by default, unless you specify the local keyword. This can cause an issue if you have an inline function that has the same name as a global function, or if you try to return functions in a closure. Here’s an example where this is an issue.

function make_counter(i)
  function incr() i = i + 1 end

  return function()
    incr(); return i
  end
end

local a, b = make_counter(0), make_counter(0)

print(a(), a(), a())
print(b(), b(), b())

This is supposed to print out 1 2 3 twice, but instead prints out 0 0 0 and 4 5 6, because the incr function ends up being shared between the two instances of the counter. Try the code out on repl.it.

The code can be fixed by simply adding the local keyword to the incr function.

function make_counter(i)
  local function incr() i = i + 1 end

  return function()
    incr(); return i
  end
end

a.func(...) vs a:func(...)

Tables in lua are just dictionaries from keys to values. The . operator can be used to get the value for certain keys. To simulate object-oriented programming you could have a table represent an object, with fields as entries. Methods would also be entries, but the value would be a function that takes in the object itself as the first argument.

local point = {
  x = 0, y = 0,
  print = function(self) print(self.x, self.y) end
}
point.print(point)

This syntax can feel redundant. Because it’s such a common pattern, Lua introduces a special : operator. You should think of a:func(...) as equivalent to a.func(a, ...).

However, this can lead to mixups if you accidentally use the wrong operator. If you wrote . but were supposed to write :, then you might get the error message attempt to index local 'self'. Otherwise, you might get some confounding type error further down the road. This is further complicated by the fact that some libraries want you to use . for objects, and thus store the variable self in a closure.

Tables are Both Lists and Dictionaries

In many languages, dictionaries and lists are separate datatypes, but in Lua these concepts are merged into one datastructure - the table. The Table is effectively a dictionary where keys and values can be anything except nil (assigning a value to be nil is the same as deleting that key, while nil simply can’t be a key). Thus one data structure can act as both a hashmap and a list, and even be both at the same time.

The Behavior of ipairs vs. pairs

pairs iterates over every key in a table, whether it’s an array-like key or a map-like key. ipairs on the other hand, has the unusual behavior that it starts at the key of 1 and then keeps incrementing the key until the value is nil. Take a look at the following cases below that shows the unusual results this behavior can create. You can run the code on repl.it.

function ipairs_print(t)
  for _, v in ipairs(t) do print(v) end
end

ipairs_print({1, 2, 3, nil, 4}) -- Prints 1, 2, 3
ipairs_print({nil, 1, 2, 3, nil, 4}) -- Prints nothing
ipairs_print({[0]=1, [1]=2, [2]=3}) -- Only prints 2, 3

The Behavior of #

The # operation is confusing in that it can actually return different results for the same table, depending on how the table was constructed. From the manual:

The length of a table t is defined to be any integer index n such that t[n] is not nil and t[n+1] is nil.

You may notice that, if your array has holes, there can be multiple such indices n, therefore the # operator can return any of those indices as a valid answer. Below is an example that showcases this unusual behavior, which you can run on repl.it.

print(#{[1]='a', [2]='a', [4]='b'}) -- Prints 4
print(#{[1]='a', [2]='a', [5]='b'}) -- Prints 2
print(#{'a', 'a', nil, nil, 'b'}) -- Prints 5

The bottom line is, unless you know for sure that your array doesn’t have holes, don’t use #.


  1. More gotchas can be found at this link - http://www.luafaq.org/gotchas.html 

View Comments

Composable, Programmatic Animations for Games

Monday, September 4th 2017


When making a game, you often have animations that play during runtime. These animations often occur in relation to other animations - either in sequence or in parallel. To solidify this concept, let’s look at the following GIF of Candy Crush.

When the user swaps two candies, two animations play in parallel - one in which the the orange candy moves to the the blue candy’s spot, and one in which the blue candy moves to the orange candy’s spot. After those two animations finish, another set of animations play in parallel. These animations include the following:

After the blue candies disappear, the candies above them fall into place.

As you can see, this single game event fires off many animations that either happen in sequence or in parallel with each other. If you have a Game loop that uses Update(dt) methods for all time-based logic, this code is going to be messy.

Enter Composable Animations

I first encountered this pattern in LibGDX, where it is simply called Actions. Using LibGDX, you can create Action objects, which can then be applied to any actor.

// Create an action that smoothly moves an object to a position.
Action action = Actions.moveTo(x, y, duration);
// Apply the action to an actor.
actor.addAction(action);

What sets LibGDX’s system apart is that you can use combinators to turn lists of actions into an action that can be further composed. For example, the code below creates an action that applies multiple actions in series.

Action action = sequence(moveTo(200, 100, 2), color(Color.RED, 6),
    delay(0.5f), rotateTo(180, 5));
actor.addAction(action);

You can use the parallel combinator to perform two actions at the same time, and the forever combinator to perform an action in an indefinite loop. Furthermore, you can simply subclass Action in order to add your own primitives that can then be composed with other primitives using the combinators.

Unfortunately, there’s a defeciency in LibGDX’s system - actions are applied to individual actors, which means that we can’t encode the dependencies between animations on different objects. Take a look at the example below from one of my projects, where two objects must animate after the first object.

To implement this, the first object applies a MoveToAction, while the other two objects apply an Action with a delay.

// The moving piece.
addAction(Actions.moveTo(worldPosition.x,
    worldPosition.y, 1.0f, Interpolation.pow3Out));

// The captured pieces.
addAction(Actions.sequence(
    Actions.delay(1.0f),
    Actions.parallel(
        Actions.moveBy(0, 2024.0f, 2.0f, Interpolation.pow3In),
        Actions.fadeOut(2.0f, Interpolation.pow3In)
    ),
    Actions.removeActor()
));

The rectify this problem, we should make Actions store the particular actor they are acting on. Then, instead of applying an action to an actor, we “play” the action, or rather “apply” it to the current Scene. To implement this pattern, I created my own library. The library is pretty simple, and in fact wraps an existing callback-based timer library. An example of an animation using my library is shown below.

local objectA = {val = 0}, objectB = {val = 0}
local action = Actions.Sequence(
    Actions.Print("Started action..."),
    Actions.Wait(3),
    Actions.Print("3 seconds passed..."),
    Actions.Parallel(
        Actions.Tween(3.0, objectA, {val=10}, 'linear'),
        Actions.Tween(2.0, objectB, {val=10}, 'linear')
    )
)

action(Timer, function() print("Continuation...") end)

This library includes an Action called Actions.Do(func) which allows you to easily interact with existing imperative animation code.

Scripting an Entire Cutscene using Composable Animations

I first applied the composable animations pattern in a prototype that I worked on. This prototype involved cutscenes that would transition seamlessly into turn-based tactical combat on a grid. Here’s a link to the code that describes the first cutscene.

Every game has unique requirements, so you can add your own Action primitives. In this particular example, I defined Actions.PlaySound and Actions.DestroyEntity, but I could even add actions for NPC barks, fade outs and much more.

View Comments

Unifying Dynamic Type Tests and Type Refinement

Sunday, September 3rd 2017


Programs often need to perform different actions based on the type of an object. Here’s an example from one of my projects, which is an Android app for the board game Hnefatafl. A class called GameState encompasses all of the infomation about a specific Hnefatafl game. Each GameState has a GameType field, which holds data specific to different types of games. The three types are GameType.PassAndPlay, GameType.PlayerVsAI, and GameType.OnlineMatch 1.

Here’s an excerpt of some code where I have to perform a different operation depending on the type of gameState.getType().

GameType type = gameState.getType();
if(type instanceof GameType.PassAndPlay) {
    GameType.PassAndPlay pap = (GameType.PassAndPlay) type;
    // ...
} else if(type instanceof GameType.PlayerVsAI) {
    GameType.PlayerVsAI pvai = (GameType.PlayerVsAI) type;
    // ...
} else if (type instanceof GameType.OnlineMatch) {
    GameType.OnlineMatch om = (GameType.OnlineMatch) type;
    // ..
} else {
    throw new UnsupportedOperationException("Unknown GameType: "
        + type.getClass().getName());
}

Because I need to access data on the type as part of my operation, I need to downcast it 2. However, there’s some obvious redundancy here - I already know the type of an object in each branch, yet I still have to downcast it to access it’s unique members. If the Java type-checker understood the instanceof operator, it could simply set the static type of type to be GameType.PassAndPlay within the associated branch.

It turns out that some languages do exactly that. Here’s an example from Kotlin. Check Kotlin’s docs on typecasts.

fun demo(x: Any) {
    if (x is String) {
        print(x.length) // x is automatically cast to String
    }
}

Here’s an example from Flow, a typechecker for Javascript. Flow understands many forms of dynamic type checks 3. See Flow’s docs on refinements.

// @flow
function method(value: boolean | Array<string> | Event) {
  if (typeof value === "boolean") {
    // value is a boolean
  } else if (Array.isArray(value)) {
    // value is an Array
    console.log(value.length);
  } else if (value instanceof Event) {
    // value is an Event
  }
}

Here’s an example from Hack, a type checker for PHP. See Hack’s docs for type refinement.

function foo(mixed $x): int {
  $a = 4;
  if (is_int($x)) { // refine $x to int by checking to see if $x is an int
    return $x + $a;
  } else if (is_bool($x)) {
    return (int) $x + $a; // know it is a bool, so can do safe cast
  }
  return $a;
}

Finally, here’s an example from C#. In C#, the refined value has to be bound to another variable name. See the docs for pattern matching.

public static double ComputeAreaModernIs(object shape)
{
    if (shape is Square s)
        return s.Side * s.Side;
    else if (shape is Circle c)
        return c.Radius * c.Radius * Math.PI;
    else if (shape is Rectangle r)
        return r.Height * r.Length;
    throw new ArgumentException(
        message: "shape is not a recognized shape",
        paramName: nameof(shape));
}

Similarities to Pattern Matching

This feature can be seen as a subset of pattern matching in functional programming languages like OCaml and Haskell. Pattern matching combines value checks, type tests, destructuring assignment, and type refinement into one construct.

Below is an implementation take in Haskell using patterm matching. take returns the first m elements of an array, dropping the rest. See the Haskell tutorial on pattern matching.

take m ys = case (m,ys) of
    (0, _)    -> []
    (_, [])   -> []
    (n, x:xs) -> x : take (n-1) xs

  1. This could have been implemented using inhertance, with different types of games being subclasses of GameState. However, I decided to go with composition over inheritance

  2. This type of operation can also be written using the visitor pattern. Unfortunately, the visitor pattern in Java is verbose, and it moves code away from the place where the behavior is used, introducing an uncessary layer of separation. The code can also be added as virtual methods to the GameType class itself, but that may not always make sense, especially if the operation is not intuitively “owned” by the GameType

  3. Flow is a gradual typechecker for an existing dynamic language. Thus Flow must understand existing forms of runtime type checks that Javascript programmers use. 

View Comments