How can one implement hot-swappable C++ modules?

Evil Engel
  • How can one implement hot-swappable C++ modules? Evil Engel

    Fast iteration times are key to developing games, much more so than fancy graphics and engines with truckloads of features in my opinion. No wonder many small developers choose scripting languages.

    The Unity 3D way of being able to pause a game and modify assets AND code, then continue and have the changes take effect immediately is absolutely great for this. My question is, has anybody implemented a similar system on C++ game engines?

    I have heard that some really super high end engines do, but I am more interested in finding out if there is a way to do it in a home-grown engine or game.

    Obviously there would be tradeoffs, and I can't imagine that you can just recompile your code while the game is paused, have it reloaded and working under every circumstance or every platform.

    But perhaps it's possible for AI programming and simple level logic modules. It's not as if I want to do that in any project in the short (or long term), but I was curious.

  • You can do something like hot-swapping modules at runtime implementing the modules as dynamic link libraries (or shared libraries on UNIX), and using dlopen() and dlsym() to load functions dinamically from the library.

    For Windows, the equivalents are LoadLibrary and GetProcAddress.

    This is a C method, and has some pitfalls by using it in C++, you can read about that here.

  • If you are using Visual Studio C++ you can in fact pause and recompile your code under certain circumstances. Visual Studio supports Edit and Continue. Attach to your game in the debugger, make it stop at a breakpoint, and then modify the code after your breakpoint. If you are to save and then continue, Visual Studio will attempt to recompile the code, reinsert it into the running executable, and continue. If all goes well the change you made will live apply to the running game without having to do an entire compile-build-test cycle. However, the following will stop this from working:

    1. Changing callback functions will not work properly. E&C works by adding a new chunk of code and changing the call table to point to the new block. For callbacks and anything using function pointers it will still be executing the old, unmodified calls. To fix this you can have a wrapper callback function that calls a static function
    2. Modifying header files will almost never work. This is designed for modifying actual function calls.
    3. Various language constructs will cause it to mysteriously fail. From my personal experience pre-declaring things like enums will often do this.

    I have used Edit and Continue to dramatically improve the speed of things like UI iteration. Let's say you have a working UI built entirely in code except that you accidentally swapped the draw order of two boxes so you can't see anything. By changing 2 lines of code live you can save yourself a 20 minute compile/build/test cycle to check a trivial UI fix.

    This is NOT a full solution for a production environment, and I've found the best solution there is to move as much of your logic as possible into data files and then making those data files reloadable.

  • Hot-swapping is a very, very difficult problem to solve with binary code. Even if your code is placed into separate dynamic link libraries, your code needs to ensure that there are no direct references to functions in memory (as the address of functions can change with a recompile). This essentially means not using virtual functions, and anything that uses function pointers do so via some sort of dispatch table.

    Hairy, constraining stuff. It's better to use a scripting language for code that needs fast iteration.

    At work, our current code base is a mix of C++ and Lua. Lua is no small part of our project, either -- it's almost a 50/50 split. We've implemented on-the-fly reloading of Lua, such that you can change a line of code, reload, and keep going. In fact, you can do this in order to fix crash bugs that occur in Lua code, without restarting the game!

  • It's definitely possible to do, although not with your typical C++ code; you'll need to create a C-style library that you can dynamically link and reload at runtime. To make this possible, the library must contain all state within an opaque pointer which can be provided to the library upon reload.

    Timothy Farrar discusses his approach:

    "For development the code is compiled as a library, a small loader program makes a copy of the library, and loads the library copy to run the program. The program allocates all data at startup, and uses one single pointer to reference any data. I use nothing global except read only data. While the program is running the original library can be recompiled. Then in one key press, the program returns to the loader and returns that single data pointer. The loader makes a copy of the new library, loads the copy, and passes in the data pointer to the new code. Then the engine continues where it left off." (original source, now available through web archive)

  • As others have said, it's a hard problem, dynamically linking C++. But it is a solved problem - you might have heard of COM or one of the marketing names that have been applied to it over the years: ActiveX.

    COM has a bit of a bad name from a developer point of view because it can be a lot of effort to implement C++ components that expose their functionality using it (although this is made easier with ATL - the ActiveX template library). From a consumer point of view it has a bad name because applications that use it, for example to embed an Excel spreadsheet in a Word document or a Visio diagram in an Excel spreadsheet, tended to crash quite a bit back in the day. And that comes down to the same issues - even with all the guidance that Microsoft offers, COM/ActiveX/OLE was/is difficult to get right.

    I will emphasise that the technology of COM is not itself inherently bad. First of all, DirectX uses COM interfaces to expose it's functionality and that works well enough, as do a multitude of applications that embed Internet Explorer using it's ActiveX control. Secondly, it's one of the simplest ways to dynamically link C++ code - a COM interface is essentially just a pure virtual class. Athough it does have an IDL like CORBA, you're not forced to use it, especially if the interfaces you define are only used within your project.

    If you are not writing for Windows, don't think that COM is not worth considering. Mozilla re-implemented it in their codebase (used in the Firefox browser) because they needed a way to componentize their C++ code.

  • (You may to know about the term "monkey patching" or "duck punching" if for nothing other than the humorous mental image.)

    That aside: if your goal is decreasing iteration time for "behavior" changes, try some approaches that get you most of the way there, and combine nicely to enable more of this in the future.

    (This will go out on a bit of a tangent, but I promise it'll return!)

    • Start with data, and start small: reload at boundaries ("levels" or the like), then work your way up to using OS functionality to get file change notifications or simply poll regularly.
    • (For bonus points and lower load times (again, decreasing iteration time) look into data baking.)
    • Scripts are data, and allow you to iterate behavior. If you use a scripting language, you now have the notifications / ability to reload those scripts, interpreted or compiled. You can also hook your interpreter to an in-game console, a network socket, or the like for increased runtime flexibility.
    • Code may be data too: your compiler may support overlays, shared libraries, DLLs, or the like. So you can now choose a "safe" time to unload and reload an overlay or DLL, whether manual or automatic. The other answers go into detail here. Do note that some variants of this may mess with cryptographic signature vecification, NX (no-execute) bit, or similar security mechanisms.
    • Consider a deep, versioned save/load system. If you can save and restore your state robustly even in the face of code changes, you can shutdown your game and restart it with new logic at the exact same point. Easier said than done, but it is doable, and it's markedly easier and more portable than poking memory to change instructions.
    • Depending on the structure and determinism of your game you may be able to do recording and playback. If that recording is just over "game commands" (think of a card game, for example), you can alter all the rendering code you want, and replay the recording to see your changes. For some games this is as "easy" as recording some starting parameters (e.g. a random seed) and then user actions. For some it is much more complicated.
    • Make efforts to reduce compile time. In combination with the aforementioned save/load or record/playback systems, or even with overlays or DLLs, this may decrease your turnaround more than any single other thing.

    Many of these points are beneficial even if you don't get all the way to reloading either data or code.

    Supporting anecdotes:

    On a large PC RTS (~120 person team, mostly C++), there was an incredibly deep state saving system, that was used for at least three purposes:

    • A "shallow" save was fed not to disk but to a CRC engine to ensure that the multiplayer games remained in lock-step simulation one CRC every 10-30 frames; this ensured no-one was cheating and caught desync bugs a few frames later
    • If and when a multiplayer desync bug occurred, an extra-deep save was performed every frame, and again fed to the CRC engine, but this time the CRC engine would generate many CRCs, each for smaller batches of bytes. In this manner, it could tell you exactly which portion of state had started to diverge within the last frame. We caught a nasty "default floating point mode" difference between AMD and Intel processors using this.
    • A normal depth save might not save e.g. the exact frame of animation your unit was playing, but it'd get the position, health, etc of all of your units, allowing you to save and resume at any time during gameplay.

    I've since used deterministic record/playback on a C++ and Lua card game for the DS. We hooked into the API we designed for the AI (on the C++ side) and recorded all of the user and AI actions. We used this functionality in game (to provide a replay for the player), but also to diagnose issues: when there was a crash or odd behavior, all we had to do was get the save file and play it back in a debug build.

    I've also since used overlays more than a few times, and we combined it with our "automatically spider this directory and upload new content to the handheld" system. All we'd have to do is leave the cutscene/level/whatever and come back in, and not only the new data (sprites, level layout, etc) would load but also any new code in the overlay. Unfortunately that's getting much harder with more recent handhelds due to copy protection and anti-hacking mechanisms that treat code specially. We still do it for lua scripts though.

    Last but not least: you can (and I have, in various very small specific circumstances) do a bit of duck punching by patching instruction opcodes directly. This works best if you're on a fixed platform and compiler, though, and because it's nearly unmaintainable, very bug-prone, and limited in what you can accomplish quickly, I mostly only use it to re-route code while debugging. It does teach you a hell of a lot about your instruction set architecture in a hurry, though.

Tags
c++ hot-reload
Related questions and answers
  • I have a lot of complex objects in a scene and im looking for an efficient way to find which object a fired bullet hits and to find the hit coordinates. It would be best if there was a lightweight library to do this. it seems that PhysX can do this, but i didnt find a way except to give every face in every mesh a physX system and that seems to become a bottleneck. thanks in advance.

  • ? My idea is to have a separate thread that compares the space difference of all the players every few seconds or so and stores references to all nearby players in the player's object for easy access... or easier way? If it can be multithreaded that would obviously be better, and the less main-thread calculation the better. (For anyone wondering, the project is in C++). ... problem is that if you had to compare all the players every time an update is sent, it would just take way too long as the number of players scales (think up to a few thousand players per map and each

  • areas of concern I have and wanted to get some opinions on the proper way others have solved them or would solve them. 1. Cyclical Dependencies When I was doing the game in C#, I didn't really have... related to the graphics library I'm using, but is more of a conceptual thing. In C#, I coupled graphics in with alot of my classes which i know is a terrible idea. Wanting to do it decoupled... sort of designs other people used to overcome them, or would use. For a little background, I started working on it in my free time last summer. I was initially making the game in C#, but about 3 months

  • ) is therefore highly wasteful. Then again, making each entity state a singleton hardly seems appropriate. What do you suggest I do? (The same can be said about game states, so whatever the solution to this is, I guess it can be applied to game states as well.) 2. The state of the entity sprite. Most of the time, whenever the state of the entity changes, the sprite that is used to represent it must... changed, then it (the representation) starts having an internal state. I would much rather have it stateless so that I require only one instance of every representation (just as before). Edit

  • for my needs. Ah and I forgot to mention that nodes can have hierarchy, but this is not a problem if the frame data is reduced. The structure instances don't get duplicated unless any value changes. When...well... I'm building the animation system of my game engine (the skeletal and skinned animation stuff), and I came to a point where I added so much functionality in the frame and node structures... know if my speculations are ok, as I don't have much experience with 3d animations yet. I want to make a well decision as any option I choose would require a lot of work to get it to render and I

  • people can play my game once finished). I'd like to have it available on many mobile platforms aswell. (because I love touch input for some reason) I do know the XNA framework pretty well...this is a post detailing my search for the most enjoyable way for a hobbyist game programmer to sweeten his free time with making a game. My requirements: I looked at Flash first, I made a couple... don't need 3D, and it is quite ...a lot of work to learn. I also don't like that it is so expensive to use for different platforms and that I can only code for it through scripting. You have to buy each

  • I been working in the animation of a 2D platformer game in C++/SDL/OpenGL, and my team and I reach the point where we need to establish that every animation of the player (Walking, Running, etc..) needs a different framerate for our concept, but I use as a guideline the Game Programming All In One as a example for the smooth animation, and in the book recommend have variables that limits the movement and the changes of frames. To clarify what I mean I have in my Sprite class these parameters: std::vector< Vector2f > delayMovementSprite; std::vector< int > frameDelayPerAnimation

  • to this (if newPos.y < y) This is all fine and dandy, but I'm wondering if there's anything that I can optimize. For example, my first thought is to store the plane's equation with the triangle's vertex information. This way all I have to do is plug in the x and z values to the equation to get the y. However, this would require adding 4 floats to every vertex of the heightmap which is a little ridiculous memory wise. Also, my heightmaps are dynamic (meaning the heights will change at runtime), which would mean changing the plane equations every time a vertex's height changes. Is there a faster way

  • After spending time today to jot down some notes regarding the implementation of walls into my tile-based game, I've suddenly realized it's not going to be as simple as I imagined before. While the current stage of my work is not even close to actually making the wall-related code, I've come up with three different ways to do it. Right now I'm unsure which one of my ideas will work best...". As you can see in the picture, a corner will basically cry for a "cap" object. I'm actually cool with it, it's not so hard to add. Hey, I already have a plan for thin columns made out of four connected