« Posts tagged Programming

Making Your Files Merge Friendly

Merging is a fact of life for most of us.  Eventually two users touch the same files and a conflict must be resolved.  For programmers merging is a daily activity, but what about the content creators?

If two artists touch the same level, have your tools programmers made enough of an effort to make merging possible and if possible as painless as can be?

Here’s a handy checklist,

1 – Use XML/JSON (or some other text based file format)

It’s slower, it’s bigger but it’s going to make merging possible.  If your level files are binary blobs merging without a custom tool just isn’t going to be possible.  Using XML or JSON are the simplest text based alternatives because there are already many libraries for reading and writing them.

2 – For XML – Give Each Attribute a Line

If you’re using XML, you should make sure to write out the files such that every element and every attribute receives its own line.  If you don’t do this it will make conflicts a lot more likely if two users touch the same object.

If you’re using JSON, having each attribute on its own line is the norm unless the library is attempting to keep the text compact.

Here’s a quick example, note that after separating the attributes the merge tool handles the conflict correctly while it fails to do so when they are on the same line.

nonewline
Figure 1 – Attributes all on the same line

newline
Figure 2 – Attributes on different lines

If you’re using .Net this can be achieved very easily by changing the default XmlWriterSettings with the NewLineOnAttribute property set to true.

C#
var settings = new XmlWriterSettings()
{
    NewLineOnAttributes = true
};
 
using (XmlWriter writer = XmlWriter.Create(filestream, settings))
{
    // Write out document...
}

3 – Maintain Order

Always write out the order of the data the same.  This one is pretty easy, the only real pitfall is to make sure your data structure and undo/redo system are working correctly.  For example, if you happen to store your objects in a list and someone removes and object at the front, but when it’s undone it’s inserted at the end of the list (Even if the UI doesn’t reflect this) this could affect your output order.

4 – Don’t Add New Objects At The End

When you go to save the entities don’t write them out in the order they were created.  This will definitely result in a merge conflict.  Because it ensures that if two users edit the file and both add an entity they’ll both show up in the same location and confuse the merge tool.

One thing you could do is to sort them based on a GUID that is stored as part of the objects data.  Sorting based on GUID ensures a lower probability of collisions occurring when two users both add objects.

Alternatively a sorting based on a string containing the machine name of the original creator of the object is another idea.  It would ensure every user touching the file would be inserting to their own section of the file instead of everyone inserting to the tail.

SWIG: Casting Revisited

A while back I wrote SWIG and a Miss, which is a post about several of the problems I’ve encountered with SWIG. At that time I didn’t have a solution for dealing with down casting – the process of casting from a base class to a more derived class.

I ended up coming up with a solution that I thought would be good to do a post on, since there doesn’t seem to be much out there on SWIG and these types of problems.

Some Background

For those unfamiliar with why it’s difficult, let me explain. When your API returns a native pointer to be wrapped, of lets say class Foo, and your function returns Foo* but the instance it returns is actually the more derived version Bar. Sadly SWIG has no way of knowing this, so the object it creates in C# would be a C# Foo class. Therefore, if you tried to cast the C# Foo to the more derived C# Bar you’d be unable to because as far as .Net is concerned the instance of Foo is just a Foo and nothing more.

Alright now that you understand the problem, lets talk about how we can solve it.

How We Can Solve It

The solution to the problem can be solved in essentially 3 ways,

1 // Walk Away

Simply don’t do anything that would require it. Don’t expose classes that users will ever need to downcast to for any reason. If this option is available to you, take it. This however means that your base class needs to support all the functionality a user will ever need, something not always possible or desirable. But if you can, do it, because once you get to script land, down casting becomes a expensive process.

2 // Manual Cast Wrapper

Write or generate C++ functions for every downcast that you would like to perform and SWIG those functions. This would allow you to take object Foo as a parameter, returns Bar, does the dynamic cast in C++ and returns the pointer to Bar.

This option is manpower intensive and is truly brute forcing a solution. It also causes potential holes where nasty aliasing occurs and can become a real problem if the source object is ever deleted. Imagine an API with a factory function that returns object Foo, but Foo is actually Bar due to the way the factory works. You need to access a Bar specific only function on the object so you downcast Foo to Bar.

Now in C++ these objects are the same object, but SWIG doesn’t know this, so two completely different C# objects are created one wrapping the native pointer to Foo, and one wrapping the native pointer to Bar. If the SWIG layer was properly authored, the factory function notes that the object returned was created with new memory and thus SWIG is responsible for deleting the object.

So we call our Bar specific function and we decide to hang onto Bar and we let Foo on the stack pass out of scope, because why hang onto it, it’s the same object as Bar…right? But Foo was responsible for managing the memory for the common native object they both point to. So the underlying native object is destroyed when the garbage collector runs next after Foo is no longer referenced and now our Bar object points to totally bogus memory.

3 // Custom Solution

The solution I ended up coming up with requires you to have a custom RTTI system that allows you to check if an object is a specific type by name.

The first step is to extend the C# wrapper for your base class with RTTI information. If you don’t have a base RTTI object this will be a bit harder. The reason we have a new m_castedSource member variable is to solve the aliasing problem. After the cast we can store the source here so that we never have to worry about the garbage collector accidentally cleaning up memory we’re still using.

C#
%typemap(cscode) YourBaseRTTIObject
%{
    internal YourBaseRTTIObject m_castedSource;
 
    public bool IsKindOf<T>()
    {
        // Use your C++ RTTI system to test if this wrapped C# object's
        // native object is a 'typeof(T).Name'
    }
%}

The next step is to create a function that can create the C# method needed to factory the C# object. We could use reflection, but that method is slow and we can do a far better job by emitting IL at runtime to specifically initialize one type and just cache those created methods.

C#
private static Dictionary<Type, Func<IntPtr, bool, object>> constructorCache =
    new Dictionary<Type, Func<IntPtr, bool, object>>();
 
private static Func<IntPtr, bool, object> CreateConstructorDelegate<T>()
{
    // We need to first grab the reflection information for the more derrived type we're casting to.
    // SWIG has protected constructors we'll be able to call that take 2 parameters, the native pointer
    // to the type and whether or not SWIG owns the memory
    ConstructorInfo ctor = typeof(T).GetConstructor(
        BindingFlags.Instance | BindingFlags.NonPublic, null,
        CallingConventions.HasThis, new Type[] { typeof(IntPtr), typeof(bool) }, null);
 
    // Lets emit some IL that will allow us to construct our type much
    // faster than using reflection.
    DynamicMethod dm = new DynamicMethod("Create", typeof(object),
        new Type[] { typeof(IntPtr), typeof(bool) }, typeof(T), true);
    ILGenerator il = dm.GetILGenerator();
    il.Emit(OpCodes.Ldarg_0);
    il.Emit(OpCodes.Ldarg_1);
    il.Emit(OpCodes.Newobj, ctor);
    il.Emit(OpCodes.Ret);
 
    // Create the delegate for the dynamic method, which will further improve the calling speed.
    return (Func<IntPtr, bool, object>)dm.CreateDelegate(typeof(Func<IntPtr, bool, object>));
}

After that we need to add a function that can perform the down cast. This is just a matter of checking if the object is actually the desired casting type. If it is we get or create the factory delegate for the type. We construct the new wrapper object and hand it the pointer for the type. (NOTE: This is only viable if the new type isn’t offset in the vtable, as would be the case with multiple inheritance).

C#
public static T CastTo<t>(YourBaseRTTIObject self) where T : YourBaseRTTIObject
{
    if (self == null)
        return null;
 
    // Check if the object is actually the type we're trying to cast to.
    if (self.IsKindOf<t>())
    {
        Type type = typeof(T);
 
        // Check if we've already cached the factory function.
        Func<IntPtr, bool, object> factory;
        if (!constructorCache.TryGetValue(type, out factory))
        {
            factory = CreateConstructorDelegate<t>();
            constructorCache.Add(type, factory);
        }
 
        // Call the factory function and set the casted source to the
        // current object.
        T castedObject = (T)factory(YourBaseRTTIObject.getCPtr(self).Handle, false);
        castedObject.m_castedSource = self;
 
        return castedObject;
    }
    return null;
}

Terraria: The Auto-Saving Server

After beating Minecraft (not really) I thought my addition to these blasted sandbox mining games was satisfied.  Sadly, I was wrong.  I started playing Terraria a few days ago and its got many more game elements than Minecraft.  It’s also wildly addicting if you’ve got a sever case of ‘Got’a’ Catch’em Alls’.  It’s also possibly the real world equivalent of Heroin Hero.

In any event, I setup a server for my friends and I to play only to discover that the server does not auto-save the world.  So if it crashes or is shutdown improperly, everything in the world resets.

So I wrote a little bit of code to fix it and decided to share.  It’s a simple bootstrapping program, you give it the file path of the Terraria server executable as the first argument and then all the other arguments to pass to it as a single string.  It then sends save commands to the server every 10 minutes.

If you want to just download a compiled version for yourself, you can download it here.

If you were to place it in the same directory as the TerrariaServer, you’d run it like this:

Text
TerrariaAutosave.exe TerrariaServer.exe "-config serverconfig.txt"

P.S. You never catch the dragon.

Extend Visual Studio!

The Pitch

I don’t want to show you how to add a command button or toolbar to Visual Studio, there are plenty of examples out there for that. I want to provide you with an example showing how writing a Visual Studio extension could benefit your studio’s workflow.

Andy Firth wrote an article on #AltDevBlogADay a few days ago on Extending the Watch Window in Visual Studio via autoexp.dat. It’s a nice introduction article to modifying the autoexp.dat file to auto-expand and format variables in the watch window.

Now, if I take a bunch of time designing a nice formatter to speed up my debugging sessions how do I share it across the company?

Well, the only way is for you to manually merge other peoples additions to the AutoExp.dat file into your own. Which inevitably leads to varying levels of adoption of the formatters and debugging times suffer as a result.

Lets fix this!

We’re going to write an extension to patch the AutoExp.dat file with one collocated with the solution file.

 

Getting Started

new_vs_packageAfter downloading the Visual Studio 2010 SP1 SDK  you’ll create a new solution using the Visual Studio Package template.  You could have also chosen the Add-in template.  Though the packages are basically just as easy to install in Visual Studio 2010 and they offer much more control should you end up needing it.

The code you’ll start out with will be a bit more verbose than this, but here’s the important bit to get started playing with different capabilities.  Visual Studio is based on a loose coupling service architecture, so to accomplish anything useful you’ll need to find the the service that handles it.  Luckily Microsoft has documented the List of Services.  If I had to pick the “main” service it would be the DTE service.  It manages the majority of the functionally you’ll be interested in playing with.

C#
public sealed class MyPackage : Package
{
    private DTE m_dte;
 
    protected override void Initialize()
    {
        // Your package is also a service container
        IServiceContainer serviceContainer = this as IServiceContainer;
        // The DTE object contains most of the goodies you'll want to play with
        m_dte = serviceContainer.GetService(typeof(SDTE)) as DTE;
        // I could also add a service of my own if I wanted
        // serviceContainer.AddService(typeof(MyService), new MyService(), true);
    }
}

 

The AutoExp.dat Patcher

Because we’ve decided to look for an AutoExp.dat patcher file beside the solution, lets hook some events so that we know when the solution is opened and closed.

C#
// Must hold a reference to the solution events object or the events wont 
// fire, garbage collection related
m_solutionEvents = m_dte.Events.SolutionEvents;
m_solutionEvents.Opened += SolutionOpened;
m_solutionEvents.AfterClosing += SolutionAfterClosing;
 
// Most of the events you'll find useful are catagoried under 
// DTE.Events...

Now when the solution is opened we’re going to scan for the patcher file in the same directory as the solution file and we’re going to use it to modify the AutoExp.dat file.

C#
void SolutionOpened()
{
    string slnFile = m_dte.Solution.FullName;
    string solutionDirectory = Path.GetDirectoryName(slnFile);
    string pathfileName = Path.GetFileNameWithoutExtension(slnFile) + ".autoexp";
    string patchFile = Path.Combine(solutionDirectory, pathfileName);
    if (File.Exists(patchFile))
    {
        string autoExpPatch = File.ReadAllText(patchFile);
        string autoExpPath = Environment.ExpandEnvironmentVariables(
            "%VS100COMNTOOLS%\\..\\Packages\\Debugger\\autoexp.dat");
 
        string blockBegin = "; BEGIN AEP [AutoExpand] (" + pathfileName + ")";
        string blockEnd = "; END AEP [AutoExpand] (" + pathfileName + ")";
 
        // What follows is a bunch string parsing code that's not relevant to the article.
        // (1) We remove the previous auto expansion block we added, if we added one 
        //     in another session.
        // (2) We insert the new auto expansion patched section from our 
        //     SOLUTION_FILE_NAME.autoexp file.
        // (3) Save the new AutoExp.dat file.
    }
}

Because our extension isn’t triggered by a command and doesn’t have a UI we need to make sure Visual Studio loads our extension from the start.  You accomplish this by adding the ProvideAutoLoad attribute to the Package class, with the right GUID identifying the time to load the extension.

C#
// --- Microsoft.VisualStudio.VSConstants.UICONTEXT_NoSolution
[ProvideAutoLoad("ADFC4E64-0397-11D1-9F4E-00A0C911004F")]
public sealed class AutoExpPatchPackage : Package
{
   //...
}

 

Results

In our test solution I’m going to start without any patch file and this is what I see when hovering over the test variable.

before_autoexppatch

After we add the patch file though and reload the solution we see something completely different.

Text
[AutoExpand]
 
TestStruct =x=<x,g>, y=<y,g>, z=<z,g>

after_autoexppatch

Voilà we’ve created a simple extension to Visual Studio that could actually improve workflow.  We can now check-in our patcher file into source control, allowing us to share our rules with all of our fellow employees.

The extension I wrote only handles the [AutoExpand] section of the AutoExp.dat file.  With a little bit of work though you could easily handle the other sections.

You could also add some other improvements such as removing the patched portions after the solution is closed to make sure there are no conflicts between versions, if you work on multiple versions of the codebase.

Another improvement would be to automatically re-patch the AutoExp.dat file if you detect a change to the patcher file while the solution is open.

 

Download

You can download the source code for the AutoExp.dat Patcher here.

 

Resources

Extend Visual Studio! @ altdevblogaday.com

SWIG and a Miss

There are so many pitfalls you’ll encounter after using SWIG for any extended period of time or with a large enough codebase.  I thought I would go over some of the more notable ones I’ve encountered creating a C# wrapper that had me sighing and doubling my caffeine intake.  That said, I do like SWIG.  If you color within the lines it works quite well and has in the end saved me more time than it has cost me.

I’m still debating if it’s worth investing time into my own cleaner/purpose built wrapper generator.  I’d like to hear some thoughts from others who have had to deal with this decision and what you chose.  If you went with writing your own, did you use any helpful libraries (like for parsing C++)?  Also, feel free to add to the this list in the comments if you’ve run into another notable problem you ran into with SWIG.

1) Consistent Generator Configuration

Using different build configurations for code generation is a bad idea.  Even though your generated code will compile using different build configurations, the #defines you set on the swig command line (-DMY_DEFINE) should be consistent in all build configurations.  Varying the generator configuration can lead to things like stale files (unless you rebuild every time), as well as an inconsistent wrapper interface.

2) Automatic Macro Expansion Is Evil

If you have a macro that does different things based upon Debug or Release SWIG will pick the one matching the -D command line define and expand the code to match that.  It won’t just insert your macro as you would if you were writing the code naturally.  The best way I’ve found to prevent this from happening is just block off the macros it’s trying to expand.

C++
#if !defined(SWIG)
    #define MYNEW ...
    #define MYDELETE ...
#endif

3) Const Correctness

C# does not have a way of mimicking C++’s const’ness.  So objects returned in C++ from a function as const Type& are treated as a regular pointer.  Allowing your wrapper user to try and change the underlying data in the object even though it’s suppose to be const.  To avoid this, you should probably remove the function entirely.  Alternatively you can treat it as a new object and generate a new C++ object on the heap and copy it there so that any edits made to it wont be reflected in the actual object.  But that has many negative sides also, like not being able to see changes in the object the reference was for.  You also create a memory leak since SWIG has no idea it has created a new object, so you need to find a way to tell swig you created new memory.  To create a wrapper class for it you can use the %typemap(out) macro like this,

C++
%typemap(out) const Vector3& %{ $result = new Vector3(*$1); %}

4) Return Reference

By default SWIG treats TYPE & as a pointer.  Which is in some sense good and in others horrible.  Like it makes perfect sense for a List class to have a function called Get(int index) that returns the TYPE& to the object in the list.  In a case like that, treating it as a pointer is fantastic.  However, suppose you have the following C++ class,

C++
class Matrix
{
    Matrix();
    Matrix& Identity();
}

Then in C# you do the following,

C++
Matrix4 wrappedMatrix = new Matrix4();
return wrappedMatrix.Identity();

Creating the Matrix4 from C# sets the flag that SWIG needs to delete the underlying native Matrix4 instance when this object goes out of scope.

Next we call Identity() which using the default C# generated code will return a pointer to the same native Matrix4 instance, but a new managed C# wrapper class will be used to wrap it.  This new C# class will have the flag set to NOT clean up the memory since it’s just holding onto a pointer and wasn’t responsible for creating it.

Finally we return the C# object created using the Identity() method, which is a different instance than the one wrappedMatrix points to, even though they both point to the same native class.  Then we return from the scope we are in, wrappedMatrix is garbage collected, the Identity() spawned matrix continues to live on in managed code, but is now pointing at a completely bogus location in memory since wrappedMatrix was responsible for deleting the native object and did so.

So how do you solve this one?  The easiest way is the same as #3, safe bet is just to not have those functions wrapped, instead provide functions that return void.  Ideally, SWIG would provide some way for me to markup the class and say,

C++
SWIG_THIS Matrix& Identity(); // SWIG_THIS doesn't actually exist.

SWIG_THIS would indicate that I’m just returning the same object, so instead of creating a new wrapper C# class when you return from the wrapped Identity() function, just return "this".

5) Customizable Allocation

There’s no standardized way of changing the allocation mechanism SWIG uses.  It will create/destroy arrays using stock malloc and free.  It will create/destroy everything else using new/delete.  The way around this is to modify the SWIG script files it uses to generate the default code. (swig\Lib\*.swg,*.i).  In particular, carrays.i for changing array allocation.  For object creation, you’ll have to modify swigs scripts for your destination language of choice.  For C# you can find them here (swig\Lib\csharp\csharp.swg).

6) Alignment Fail

SWIG and aligned data types don’t get along together.  Presumably you have your own allocator and have either A) Have overridden global new/delete, B) Created a common base class with new/delete functions, or C) Created a macro to use on every class you want to override new/delete on.  If you have then in the case of object creation, you’re safe.  Though you may want to replace SWIGs use of new/delete anyway with your new/delete macros as I am sure you’d like to make sure you track exactly where those allocations came from.  If you have any arrays SWIG creates, you’ll definitely need to replace their usage of malloc and free with one going through your allocator so that you can handle alignment.

7) Nested Types

SWIG doesn’t handle nested data types.  I haven’t found a way around it, you just have to pull them out of the parent class if you want them wrapped.

8) Where Are The Out/Ref Parameter Flags

SWIG doesn’t map method parameters that are passed by (non-const) reference as "out" or "ref" parameters in C#.  The way around this is to do the following,

C++
%include "typemaps.i"
 
%define %TypeOutParam(TYPE)
    %apply TYPE& OUTPUT { TYPE& };
%enddef
 
%TypeOutParam(bool)
%TypeOutParam(int)
%TypeOutParam(float)
...

That will make all instances of those types being used in the context void MyMethod(int& x, int& y) would be mapped to void MyMethod(out int x, out int y) in C#.  Alternatively if I had done,

C++
%include "typemaps.i"
 
%define %TypeRefParam(TYPE)
    %apply TYPE& INOUT { TYPE& };
%enddef
 
%TypeRefParam(bool)
%TypeRefParam(int)
%TypeRefParam(float)
...

The method void MyMethod(int& x, int& y) would be mapped to void MyMethod(ref int x, ref int y) in C#.

9) IntPtr == void*

For some unknown reason SWIG does not map void* to anything useful in C#.  So I came up with the following to map it to the IntPtr struct in C#.

C++
%typemap(ctype)  void * "void *"
%typemap(imtype) void * "IntPtr"
%typemap(cstype) void * "IntPtr"
%typemap(csin)   void * "$csinput"
%typemap(in)     void * %{ $1 = $input; %}
%typemap(out)    void * %{ $result = $1; %}
%typemap(csout)  void * { return $imcall; }

10) string[] –> char**

For C# string[] is not mapped to char**.  So here is how you map it,

C++
%typemap(ctype) char** "char**"
%typemap(imtype) char** "string[]"
%typemap(cstype) char** "string[]"
 
%typemap(csin) char** "$csinput"
%typemap(csout, excode=SWIGEXCODE) char**, const char**& {
    int ret = $imcall;$excode
    return ret;
  }
%typemap(csvarin, excode=SWIGEXCODE2) char** %{
    set {
      $imcall;$excode
    } %}
%typemap(csvarout, excode=SWIGEXCODE2) char** %{
    get {
      int ret = $imcall;$excode
      return ret;
    } %}
 
%typemap(in) char** %{ $1 = $input; %}
%typemap(out) char** %{ $result = $1; %}

11) Operators

Operators are not wrapped by default.  There is probably a way to map them to operators in C#, but I went with another route. You can rename them so that they are, for example

C++
%rename(Add) Vector3::operator +;

12) Memory Lifecycle Management

SWIG has this internal flag in generated classes that tells it if it needs to clean up the memory for the native object when the instance is garbage collected by .Net.  If you have pesky life-cycle management requirements in your system you’ll need to be able to control this flag on the fly.  The way I’ve found to do this is with the %typemap macro,

C++
%typemap(cscode) YOUR_CLASS
%{
    public bool IsLifecycleManaged
    {
        get { return swigCMemOwn; }
        set { swigCMemOwn = value; }
    }
%}

Now you can control on a per instance basis whether or not .Net garbage collecting the object will clean it up.

13) Callbacks

If you need to do callbacks use the "%feature("director")".  The documentation explains it more in depth, but it wasn’t clear to me at first how to do it until I learned about directors.

14) Extension Macro

You can extend/add functionality to the C++ class prior to the wrapper code getting generated using the "%extend" macro.  The extension code is added as a C function though, so you might wonder, how do I access the class that this function is supposed to be part of.  The code SWIG generates refers to the ‘this’ pointer as the variable ‘self’ so you can access anything public on the class by just doing

C++
self->...

15) Unfriendly Template Class Names

SWIG generates unfriendly names for templated classes.  The best way around this is to use the %template macro.  It works like this,

C++
%template(ListFloat) List<float>;

16) Tracking New Memory

If a function creates a new object and returns it, you need to tell SWIG this, otherwise you’ll have a memory leak.  By default SWIG does not destroy any object it wraps unless it created a temporary object in the heap for a value type on the stack.  The way you inform swig a method returns a new object is by doing,

C++
%newobject MyFactory::Create;

17) Casting

As it stands I don’t have a good solution to this one. I thought I should mention it though, because it has been a real thorn in my side. If you have a method in C++ that you wrap and you return a base class like BaseObject*. SWIG will wrap that inside a BaseObject C# class. Now all the other SWIG wrapped classes that derived from BaseObject will exist and will similarly inherit from a C# BaseObject.

However, attempting to cast your new C# BaseObject into anything other than its parent classes will result in an exception. Because SWIG has no way of knowing the true type of the BaseObject* it returned from the function. So the C# object it creates is going to be BaseObject and not the actual C# wrapper class for the native types true type.

SWIG and a Miss @ altdevblogaday.com