Ingrater's 3D Blog

Compiling MinGW-GDC on Windows

admin — Sun, 10 Mar 2013 18:05:50 +0000

In the past days is struggeled compiling MinGW-GDC on Windows, so here is a full list of instructions what needs to be done:

Install msys-git aviable from http://git-scm.com/
Start a git console and type
```
git config core.autocrlf
```
if this prints “true” you need to run
```
git config --global core.autocrlf false
```
For more information see: https://help.github.com/articles/dealing-with-line-endings
cd to the folder where you want the GDC sources stored and then do
```
git clone git://github.com/venix1/GDC
git checkout mingw
```
Open a random text file in the GDC folder and make sure it actually has LF (Linux-Style) line endings. If not you did something wrong.
Download the mingw-get installer from www.mingw.org
During installation make sure you pick the “Download latest repository catalogues” option.
When you are asked which components to install pick:
- C Compiler
- C++ Compiler
- MSYS Basic System
- MinGW Developer ToolKit
Now start a msys command shell

Execute

mingw-get install msys-rsync
mingw-get install msys-wget
mingw-get install msys-unzip

Create a folder named “crossdev” directly in the root of your C: drive. (Make sure you have at least 4 gigs of space left)
Execute
```
notepad /etc/fstab
```
And make sure it has the follwing contents (assuming you installed MinGW to C:\MinGW)
```
C:\MinGW\   /mingw
C:\crossdev\ /crossdev
```
Now cd to the directory where you checked the sources out to and execute
```
./build-tdm-gdc.sh
```
When the script is done the compiled gdc will be stored in C:\crossdev\gdc64\v2\release

Known Issues

The script gets stuck after printing “Checking if the compiler has the remainder bug…”
If this happens just kill the conftext.exe with the taskmanager

Compile errors in options.c
GDC/gcc/d/lang.opt has CRLF line endings, make sure you have LF line endings in the whole GDC source tree

Compile errors while building libppl containting “itimervalue”
Once again CRLF line ending issues. Make sure that all the source files you checked out have LF line endings.

Runtime code reloading in D, part 1

admin — Sun, 30 Dec 2012 12:30:03 +0000

One of the biggest problems in game developement are turnaround times. Tournaround time is the time you have to wait until you can see what your change actually did. Shorter turnaround times improve productivity as you don’t spend that much time waiting for changes to become visible. Also shorter turnaround times support the creative process as humans have huge dificulties connecting an action and a result that happens with a huge delay. For mappers, 3d artists, effect artists and sound designers the turnaround times in modern 3d engines are almost zero. A lot of work is put into their tools to keep turnaround times to a minimum. Programmers however usually have very long turnaround times. Mostly this is due to the fact that compile times of the C++ language, which is mostly used in game developement, can go into multipe minutes. This problem is almost solved in the D programming language, as compile times are really quick and it usually only takes a few seconds to compile a project. But after making changes to the code you still have to restart the game, wait for it to load all the resources and then fly/walk to the spot in the level you are currently interrested in. I got really tired of this process so I decided to build a runtime reloading for certian parts of my own little game engine.

Game simulations in general have nice properties for runtime code reloading. The simulation is performed in steps, and after each step there usually is a synchronisation point. At that point you exactly know what part of your code is currently executing, so this is ideal to patch other parts of the code as it is currently not executing. For a start I wanted to build a simple system that allows reloading modified algorithms but does not support changes in the structure of the data. This has the advantage that objects remain at the same position on the heap and only the vptrs have to be patched. This makes patching easier for a start.

Runtime type information

To be able to patch existing object instances you will usually need some kind of runtime type infomration (RTTI) system. Usually you need to know

Where are the vptrs inside the class
Which members point to other objects

The first point is already covered by the D’s TypeInfo instances that exist for every type at runtime. To have information about the members I wrote my own little RTTI system that uses the RTInfo template inside object_.d. This template is instanciated for every type that is used during compilation and thus is ideal to generate RTTI information. At runtime the generated information can be accessed via the “rtInfo” member of the respective “TypeInfo” object. At some point this template will be used for a percise garbage collector but at the time of this writing it is unused (dmd 2.061). Additionally I’m not using D’s GC so even if it is used at some point, I can replace it with my own version because I won’t need percise GC data.

The RTTI info stored for each member looks as follows:

struct thMemberInfo 
{
  string name;
  TypeInfo type;
  size_t offset;
  thMemberInfo[]* next;
}

I use D’s compile time function execution (CTFE) to generate the code which creates the RTTI info. Note the “static if” statements that avoid cyclic references.

string makeRttiInfo(T)()
{
  string result = "[thMemberInfo(\"" ~ T.mangleof ~ "\", typeid(T), instanceSize!T, null)";
  size_t i=0;
  foreach(m; __traits(allMembers, T))
  {
    static if(is(T == struct))
    {
      static if(is(typeof(__traits(getMember, T, m).offsetof)))
      {
        size_t offset = __traits(getMember, T, m).offsetof;
        static if(is(typeof(__traits(getMember, T, m)) == struct))
          result ~= ",\nthMemberInfo(" ~ m.stringof ~ ", typeid(typeof(T." ~ m ~ ")), " ~ formatOffset(offset) ~ ", &RttiInfo!(typeof(T." ~ m ~ ")))";
        else
          result ~= ",\nthMemberInfo(" ~ m.stringof ~ ", typeid(typeof(T." ~ m ~ ")), " ~ formatOffset(offset) ~ ", null)";
      }
    }
    else
    {
      static if(__traits(compiles, mixin("(T." ~ m ~ ").offsetof")))
      {
        size_t offset = mixin("(T." ~ m ~ ").offsetof");
        static if(is(typeof(__traits(getMember, T, m)) == T) || !is(typeof(__traits(getMember, T, m)) == struct))
          result ~= ",\nthMemberInfo(" ~ m.stringof ~ ", typeid(typeof(T." ~ m ~ ")), " ~ formatOffset(offset) ~ ", null)";
        else
        {
          //pragma(msg, "Reference for " ~ T.stringof);
          result ~= ",\nthMemberInfo(" ~ m.stringof ~ ", typeid(typeof(T." ~ m ~ ")), " ~ formatOffset(offset) ~ ", &RttiInfo!(typeof(T." ~ m ~ ")))";
        }
      }
    }
  }
  return result ~ "]";
}

For the class “RigidBody”:

abstract class IRigidBody
{
public:
  Position position;
  Quaternion rotation;
  vec3 velocity;
}

class RigidBody : IRigidBody
{
  CollisionHull m_collision;
  float m_inverseMass;

  public:
    float remainingTime;
}

This will produce the following RTTI information:

[thMemberInfo("C7physics9rigidbody9RigidBody", typeid(T), instanceSize!T, null),
thMemberInfo("m_collision", typeid(typeof(T.m_collision)), 80, null),
thMemberInfo("m_inverseMass", typeid(typeof(T.m_inverseMass)), 84, null),
thMemberInfo("remainingTime", typeid(typeof(T.remainingTime)), 88, null),
thMemberInfo("position", typeid(typeof(T.position)), 8, &RttiInfo!(typeof(T.position))),
thMemberInfo("rotation", typeid(typeof(T.rotation)), 52, &RttiInfo!(typeof(T.rotation))),
thMemberInfo("velocity", typeid(typeof(T.velocity)), 68, &RttiInfo!(typeof(T.velocity)))]

The first line of the RTTI info contains the mangeled name of the type this information was generated for, the TypeInfo object for the current type and the size for the current type. The manageld name can be used as a unique identifier when building lookup tables for type information. The remaining entries of the array contain information about all members (including the members of the base classes) in the following order:

Member name
member type info
member size
pointer to the RTTI information for this member

The pointer to the RTTI information for each member is a workaround for a strange bug I encountered. For POD structs the rtInfo template will be instancated, but for some reason you can not query the results at runtime. It is possible that the linker stripps the information away but I could not yet investigate this issue further. If I can not query the RTTI information at runtime I fallback to the RTTI info pointer stored in the “next” member of thMemberInfo.

To trigger the generation of the RTTI info the following code is used:

template RttiInfo(T)
{
  __gshared RttiInfo = mixin(makeRttiInfo!T());
}

And the RTInfo template inside object_.d and object.di looks like this:

template RTInfo(T)
{
	version(RTTI)
	{
		static if(is(T : TypeInfo) || !(is(T == struct) || is(T == class)))
		{
			enum RTInfo = null;
			//pragma(msg, "ignoring " ~ T.stringof);
		}
		else
			enum RTInfo = &RttiInfo!T;
	}
	else
	{
		enum RTInfo = null;
	}
}

As soon as you pass the “RTTI” version identifier to the compiler all types will have RTTI generated for them. As you might notice I only generate RTTI for structs and classes with the exception of classes that are dervived from TypeInfo. TypeInfo classes are ignored because you don’t really need reflection data for the refelection data .

Plugin system

To be able to actually reload code at runtime, I did split of the part of the code that should reloadable into a own dll. This was quite compilcated as D does currently not support shared dlls and I had to make small modifications to the runtime to allow for this feature. I might write another article about this in the future.
All calls into this dlls are done over D interfaces, which means that they are virtual and always go through a vtable. By patching the vptrs all these calls can be redirected to the new code. Each plugin also has a few “roots”. For the system to work, all objects that have been allocated by a plugin should be findable by wakling down the object hirarchy from the roots of a plugin.

Patching

To be able to recompile and reload the dll at run time, the dll and its debugging info (.pdb file) is copied to a temporary subfolder before it is loaded with a LoadLibrary call. To detect modifications to the plugin dll ReadDirectoryChangesW is used. (see my standard library on github). After a modification has been detected the reloading works as follows:

Wait for the end of the current simulation cycle
Copy the dll and pdb file to a new subdirectory
Load the new dll
Walk the type hirarchy of the new dll and build a dictionary of all type information objects
Iterate over all objects allocated by the old dll and patch them
Continue the simulation

The old plugin remains loaded to keep references to static or TLS memory valid. Each plugin has a small proxy allocator that keeps a list of all allocations done by a plugin so that iterating over them becomes possible. Refernces to objects that are not allocated by the plugin that is currently reloaded remain untouched.

Building a type info dictionary

The following function runs down the type hirarchy to build a hashmap with all type info objects for the newly loaded plugin. Please note that it does not operate on actual object instances but merely on the pure type information. This gets a lot easier with a small extension to D’s TypeInfo classes that has been submitted as a pull request to the regular druntime at the point of this writing: druntime pull request 370. This exstension adds a “type” property to all TypeInfo classes which returns the type of the TypeInfo object as a enum value. This has the advantage that it works flawlessly across dll bounadries which would not be possible with the “is” operator. This is due to the fact that currently the TypeInfo instances are duplicated for each dll, which results in different addresses for instances of the same TypeInfo object across dll boundaries.

struct ScanPair
{
  void* addr;
  TypeInfo type;
}

private final void BuildPluginTypeInfo(IPlugin plugin, Hashmap!(string, const(thMemberInfo)[], StringHashPolicy) types)
{
    ScanPair[10] roots;
    size_t numRoots = plugin.GetScanRoots(roots);
    foreach(ref root; roots[0..numRoots])
    {
      auto rttiInfo = getRttiInfo(root.type);
      auto context = BuildTypeInfoContext(types);
      context.buildInfo(rttiInfo);
    }
}

static struct BuildTypeInfoContext
{
  Hashmap!(string, const(thMemberInfo)[], StringHashPolicy) types;

  void buildInfo(const(thMemberInfo[]) rttiInfo)
  {
    if(rttiInfo.length > 0)
    {
      //avoid endless recursion
      if(types.exists(rttiInfo[0].name))
        return;
      types[rttiInfo[0].name] = rttiInfo;

      //iterate all members
      foreach(ref info; rttiInfo[1..$])
      {
        if(info.next !is null)
          buildInfo(*info.next);
        else
        {
          auto tt = info.type.type;
          //For compond types use the next type
          if(tt == TypeInfo.Type.Array || tt == TypeInfo.Type.StaticArray || tt == TypeInfo.Type.Pointer)
          {
            buildInfo(getRttiInfo(info.type.next));
          }
          else
          {
            buildInfo(getRttiInfo(info.type));
          }
        }
      }
    }
  }
}

Patching the objects

After a dictionary of all type info objects has been build the runtime objects can be patched. All runtime objects are iterated by running down the object hirarchy starting at the plugin roots. For each plugin it is checked if it was allocated by the object to be patched, and if yes its new type information is recieved from the hashmap and all vptrs are patched. This is quite easy because D’s TypeInfo system already provides enough information to locate all vptrs. The memory layout of a D object is as follows (tested with DMD 2.060, no garantuees for correctness):

We are only interrested in the location of the vptrs. The object vptr is always at the very start of the instance so we can easily patch that.
The location of all interface vptrs can obtained from the “interfaces” member of the TypeInfo_Class type information. Additionally the inheritance chain has to be walked until the very top is reached to find all interface vptrs. The new values for all vptrs can be obtained from the TypeInfo “init” property. Which is a void[] containing the values a class instance should be initialized with. This will result in the following code for patching interface vptrs.

//Now patch all the interface vtbls
for(TypeInfo_Class cur = newType; cur !is null; cur = cur.base)
{
  foreach(ref i; cur.interfaces)
  {
    *cast(void**)(addr + i.offset) = *cast(void**)(newType.init.ptr + i.offset);
  }
}

The code for patching a entire object hirarchy looks currently like this:

final IPlugin ReloadPlugin(const(char)[] pluginName)
{
  IPlugin oldPlugin;
  size_t pluginIndex;
  foreach(size_t i, ref info; m_loadedPlugins.toArray())
  {
    if(info.name[] == pluginName)
    {
      oldPlugin = info.plugin;
      pluginIndex = i;
      break;
    }
  }
  if(oldPlugin is null)
  {
    throw New!RCException(format("Can't reload plugin '%s' because it is not yet loaded", pluginName));
  }

  HMODULE hModule = CopyAndLoadPlugin(pluginName);

  PluginInitFunc initFunc = cast(PluginInitFunc)GetProcAddress(hModule, "InitPlugin");
  PluginDeinitFunc deinitFunc = cast(PluginDeinitFunc)GetProcAddress(hModule, "DeinitPlugin");
  PluginGetFunc getFunc = cast(PluginGetFunc)GetProcAddress(hModule, "GetPlugin");
  if(initFunc is null || getFunc is null || deinitFunc is null)
  {
    throw New!RCException(format("Loading plugin '%s' failed because %s%s%s", pluginName, 
                                  (initFunc is null) ? "InitPlugin entry point not found " : "",
                                  (getFunc is null) ? "GetPlugin entry point not found " : "",
                                  (deinitFunc is null) ? "DeinitPlugin entry point not found " : ""));
  }

  if(!initFunc(g_pluginRegistry, null))
  {
    throw New!RCException(format("Initializing plugin '%s' failed", pluginName));
  }

  IPlugin newPlugin = getFunc();
  if(newPlugin is null)
  {
    throw New!RCException(format("Getting reloaded plugin '%s' failed", pluginName));
  }

  //First build the type info for the new plugin
  auto types = New!(Hashmap!(string, const(thMemberInfo)[], StringHashPolicy))();
  scope(exit) Delete(types);
  BuildPluginTypeInfo(newPlugin, types);

  //Now patch the roots
  ScanPair[10] oldRoots;
  ScanPair[10] newRoots;
  size_t numOldRoots = oldPlugin.GetScanRoots(oldRoots);
  size_t numNewRoots = newPlugin.GetScanRoots(newRoots);
  if(numOldRoots != numNewRoots)
  {
    throw New!RCException(format("Error reloading plugin '%s': number of roots does not match", pluginName));
  }

  for(size_t i=0; i 
For my current physics simulation plugin this is already enough to reload it at runtime. Usually I have two visual studio instances open at the same time. One running the main application in the debugger. I use this instance to debug issues and make changes. When the changes are done I switch to the second visual studio instance which only has a solution with the physics simulation plugin in it. This visual studio instance is only used to recompile the plugin. When I then give the focus back to the game application, it checks for modified files, reloads the plugin because it was modified and then I can see the results of my changes right away.
There are obviously many issues left to be addressed. The next one I will focus on is reloading a plugin, which changed the structure of some of its classes. I plan to serialize all class instances and deserialize them again with the new plugin code. As the address of all these instances will change all references to these instances have to be found and patched, witch will be the hard part of it. 
I recognize that the solution presented here only works with all the preconditions I listed and might not be aplicable for other fields of software developement, which is a huge downside. Despite this, I hope that it holds some interresting information for other fields of software developement too.

Real World Comparison, GC vs. Manual Memory Management

admin — Wed, 05 Sep 2012 10:54:49 +0000

During the 4th Semester of my studies I wrote a small 3d spaceship deathmatch shooter with the D-Programming language. It was created within 3 Months time and allows multiple players to play deathmatch over local area network. All of the code was written with a garbage collector in mind and made wide usage of the D standard library phobos. After the project was finished I noticed how much time is spend every frame for garbage collection, so I decided to create a version of the game which does not use a GC, to improve performance.

In a pc game you usually want to achive 60 FPS (frames per second). That means you
have 16.6 ms time to simulate and render a single frame. Also you have to prevent big variations in frame time, as this will lead to visual stuttering or other issues.

I created three version of the game for comparsion:

GC Version compiled with DMD: The original version of the game. Compiled with dmd 2.058 (-O -release -noboundscheck -inline). Not all memory is managed by the garbage collector. Large blocks of memory that only contain data and no pointers are allocated manually to improve garbage collector performance. In this version the GC is run manually once every frame, otherwise the frame times would vary to much, as you would get collection times of multiple seconds, which is not acceptable for games.
GC Version compiled with GDC: Same as above just compiled with 2.058 GDC eqivalent. (-fno-bounds-check -frelease -O -finline-small-functions -findirect-inlining -fpartial-inlining -fpeephole2 -fregmove)
Manually Memory Managed Version compiled with DMD: I throw away most of phobos and wrote my own replacements for it with different interfaces, as the phobos interfaces are usually not suitable for manual memory management. Small parts of phobos could be reused, for example std.traits. Also I made quite some changes to druntime. I added a reference counting mechanism to druntime to make threads reference counted. Also I added a memory tracker which would track and report memory leaks on program ending. Also all parts of druntime that did leak memory during developement have been fixed. For example I implemented a non leaking cache friendly hashmap. This version of the game uses manual memory management most of the time. If manual management is not feasable reference counting is used instead.

Other then the memory management code, the code of the GC version and the manual memory management version are exactly the same. This is the ideal real world comparison for GC vs. manual memory management.

Results

DMD GC Version: 71 FPS, 14.0 ms frametime
GDC GC Version: 128.6 FPS, 7.72 ms frametime
DMD MMM Version: 142.8 FPS, 7.02 ms frametime

GC collection times:

DMD GC Version: 8.9 ms
GDC GC Version: 4.1 ms

The astonishing thing here is, that the manual managed version compiled with dmd is still faster then the highly optimized GC version GDC generates. I could not compile my manually managed version with GDC yet as druntime for GDC has a completely different folder structure and quite some modifications.

Most of this performance improvements come from the fact that the garbage collector does not eat any time. But also if you write code with a GC in mind you often do not think about the consequences when allocating memory, because the GC will deal with it. This often leads to highly imperformant code, the best example is the comparison of TypeInfo objects in druntime, which is described below.

Biggest Performance and Memory Leaking Issues in D 2.0:

Here is a list of the most severe problems I found during developement:

Comparision of TypeInfo objects in druntime is done by building two strings and then comparing those two strings. This will always leak memory and do a lot of unneccesary allocations which are a performance bottleneck. I reworte the comparison so it does not allocate a single byte.
Calls to the druntime invariant handler are emitted in release build also and there is no way to turn them off. Even if the class does not have any invariants the invariant handler will always be called, walk the class hirarchy and generate multiple cache misses without actually doing anything.
The new statement will not free any memory if the constructor throws a exception. So you are forced to replace both the new and the delete statement. But you can not replace the new statement with similar nice syntax especially for inner classes and arrays.
Inlining of DMD. Inlining of DMD seems to be very minimal. Most of my overloaded operators are not inlined by dmd at all.
Array literals. They always allocate, even if they don’t have to. For example when asiging to a fixed size array, or when passing to a function with a scope parameter.
D-Style variadic functions. Same as array literals, because internaly they are rewrote to one. Especially for these kind of functions I don’t see any reason why they should allocate on the heap.

Currently only my modifications to druntime and phobos are aviable on github. But the full sourcecode for my small standard library and the game wil follow soon. All this has been done with dmd 2.058 and the druntime and phobos state of 2.058. But now that I’m done I will update to 2.060 soon.

druntime (build with make -f win32nogc.mak, currently only works on windows)
phobos (build with make -f win32nogc.mak, currently only works on windows)
thBase (my standard library)

Update:
I found a piece of code that did manually slow down the simulation in case it got to fast. This code never kicked in with the GC version, because it never reached the margin. The manual memory managed version however did reach the margin and was slowed down. With this piece of code removed the manual memory managed version runs at 5 ms which is 200 FPS and thus nearly 3 times as fast as the GC collected version.

Alienfx winamp plugin update

admin — Tue, 20 Sep 2011 18:50:31 +0000

As no one picked up the developement on the winamp plugin after I released the sourcecode and new alienware laptops keep coming out every year, so I decided to put a litte bit more work into the winamp alienfx plugin. With this update users are able to add support for new laptops themselfs by editing a xml file. I will explain here how to create the xml file.
To download the plugin go to the original post: AlienFX Winamp plugin

To follow this guide you need a Alienware laptop or PC with a AlienFX device and some xml editing skills.
First you need to find out the product id of your alienfx device. To do that open the windows hardware manager. Go to start -> control panel -> system -> (top left) device manager.
In the device manager open the “input devices (Humand Interface Devices)” group and right click onto the first “USB-input device” in the list. Go to properties.

In the properties window go to details and select “Hardware-IDs” from the dropdown list.

The shown lines should start with “USB\VID_187C”.

If they do thats the AlienFX device. If not you have to try the other usb input devices. After the VID_187C you can see the product id “PID_0511″. You can ignore the PID_ and the zero, that gives us the product id 511 in this case. Because this is hex notation you need to add 0x to the start of the product id, that gives you in this case the final product id 0×511.
Write down the product id you will need again.
Now download the AlienFX leds tester from here: AlienFX leds tester
Start the program and follow the instructions until it asks you if you want to test a custom device.
Press Y and enter to answer with yes. Now type in the product id you just wrote down.
Next it is going to ask you if it is a new alienfx device. If the winamp plugin does not support your device by default, it is most likely a new device, so press y and enter. Now follow the instructions until the program ends. If none of the alienfx leds on your laptop changed while the program ran, it did not work. If everything worked fine you should end up with a results.log file in the same directory that contains the leds tester. It should look something like this:

Old Alienfx Device
Model description: Area-51 m15x
Leds still on: power led
Led 0x1: touchpad
Led 0x20: light pipe
Led 0x80: alienware logo
Led 0x100: alienhead
Led 0x10000: touchbar

Keep this file open you are going to need it.
No go to your “appData/Winamp/plugins” directory (“C:\Users\Your Username\appdata\Roaming\Winamp\Plugins” for Vista and 7)
Create an empty textfile and name it “alienfx_vis_devices.xml” (make shure you have “hide known file endings” disabled in the windows folder options). Edit this xml file and paste the following template into it:

Enter the name of your laptop into the name attribute. If you have a new alienfx device (anything newer then the Area-51 series) enter 2 for the protocol version, otherwise enter 1. Usually you want to enter 2 for the protocol version as all laptops with the old protocol version are supported by default.
for the productId enter the product id you just found out.
Now you need to add the following line inside the devices node for each led you have on your laptop:

For name enter the name of your led, and for code enter the code for that led. You can find the code in the results.log file you should still have open. The format in the results.log is: “Led (led code): (led name)”.
When you are done editing the xml file it should look like this:

Fire up winamp, start the plugin. If you made any mistakes there should be error messages telling you what you did wrong. Fix the erros and try again. If everything started without errors you can go to the config dialog and configure the leds to respond to certain beats, and test the plugin. If you have a working xml for your laptop be nice and upload it to notebookrecords.com and/or send me a e-mail with the xml attachted to it. It is also possible to enter multiple devices into this xml file. Just add additional “device” nodes to the xml file. The plugin will try all devices in the xml file upon start, and use the first one it is able to find.

Suggestions for the D 2.0 Programming Language

admin — Sat, 27 Aug 2011 17:09:07 +0000

First of all don’t take this article to negative. I really like the D programming language and often when I have to go back to C++ coding im sitting in front the code thinking “Oh this would be so much easier in D” or “This could be solved much cleaner in D”.
I recently completed 3 Projects using the D 2.0 programming language:
- A implementation of the Light Propagation Volumes algorithm
- A 3d multiplayer cross platfrom (windows / linux) space shooter
- A lisp interpreter (with almost all Scheme features)

The two 3D projects have been done using OpenGL and are very performance critical. You only get to spend 16ms every frame until it has to be completely rendered otherwise you won’t reach fluent framerates. The lisp interpreter was performance critical too, but more of a test how good D is usable as a systems programming language.

1. Assert
Assert throwing a exception is very unhandy for debugging a application, especially when some of the stacktracing and stack unwinding happend before the debugger actually gets the signal. This often lead to not beeing able to see where the assert actually triggered. Usually I had to look it up in the console output, set a breakpoint at the perticular assert and rerun the application which added quite some turnaround time. Using a custom assert is also not possible because the compiler will complain about certain “not reachable” problems. Only the builtin assert is recognized by the compiler and there is no way of flagging another function as assert. Either the builtin assert should be configureable so that it triggers a breakpoint before throwing the exception, or there sould be a way to built a custom assert because something like this feels wrong:

bool bar(){
  switch(foo){
    case 1:
      ...
    default:
      ... 
  }
  myAssert(0,"not reachable");
  assert(0,"not reachable");
}

2. Emplace requiring the exact constructor argument types
Using D with manual memory management is a very nice thing, besides not beeing able to use phobos then and having to be very carefull when handeling arrays. But as the new and delete operator are deprecated one problem within phobos should be adressed:
When using emplace to create a struct or class emplace is not aware of the consturctor overloading rules and unless given the exact same types as the constructor it is not able to construct the object:

class foo {
  this(int* addr){ ... }
}

void[__traits(classInstanceSize,foo)] mem;
emplace!bar(mem,null); //this will not compile

3. Structs not having identity
Structs in D don’t have identity, meaning they can be copied around by the compiler whenever he wants. This was especially a problem when implementing a custom GC for the Lisp interpreter. Because I wanted to implement a Baker GC which requires to know the addresses of all references on the stack I wanted to do something like this:

struct Ref(T) if(is(T == class)){
  T m_Ref;
  this(T pRef){
    m_Ref = pRef;
    myGC.addRoot(&m_Ref);
  }

  ~this(){
    myGC.removeRoot(&m_Ref);
  }
}

Unfortunaltey when the destructor gets called m_Ref can have a completely differend address and therefore can not be deregistred at the garbagecollector. I ended up adding a assert into the desturctor telling me when the address changed and not using any exception handling because I figured that the struct copying then will not happen. But that is not a acceptable solution for the problem. Adding one level of indirection to every reference is also not acceptable for performance reasons. Now I’m not saying that structs should have identity, there should just be just a way to get notified when a struct gets moved. Maybe a move constructor, something like

struct Ref(T){
  ...
  this(this,void* oldAddr){
    myGC.removeRoot(oldAddr);
    myGC.addRoot(&m_Ref);
  }
}

This feature coupeled with the discussed -nogc flag that would replace every reference in the sourcecode with a Ref template inside _object.d would actually allow implementing GCs for the D language that are not based on the mark & sweep approach, because they could track references. (There needs to be a different solution for references inside classes and structures of course)

4. Implicit constructors
A feature I actually like sometimes about C++ is implicit constructors. They often cause a lot of problems but when used correctly they can be very usefull. Again given the “lets implement a different GC” task:

void foo(Object bar){
  ...
}

foo(new Object());

If you now want to replace all refrences with a template it would look something like this:

void foo(Ref!Object bar){
  ...
}

As new does return a Object and not a Ref!Object this will not compile. If there would be a way to define implicit constructors
this problem could be solved:

struct Ref(T){
  implicit this(T pRef){
    ..
  }
}

This would introduce a new keyword to the D programming language doing the exact opposide of C++ explicit. You have to delcare implicit constructors not explicit ones. This would be the same “I know what I am doing” approach D usually uses.

5. Shared
Maybe I’m not getting this keyword, but I consider shared not beeing usable. I would highly appreciate a offical document about how shared sould be used, and not only in some small mini example using message passing, but in a realworld performance critical example.
Say for example I have a Stack container:

class Stack(T){
  ...
  void push(T el){ ... }
  T pop(){ ... }
  bool empty(){ ... }
}

And now I need a threadsafe version of it. Should be quite easy, especially because D supports synchronized classes:

synchronized class ThreadsafeStack(T){
  Stack!T m_Stack;
  void push(T el){ m_Stack.push(el); }
  T pop() { return m_Stack.pop(); }
  bool empty(){ return m_Stack.empty(); }
}

This will not compile! Because synchronized classes are implicitly shared and shared is transitive, m_Stack will be shared and thus calling the methods of Stack!T is not possible because they are not shared. This transitivy basically kills you everywhere. If you have a shared class basically everything in the hirarchy it accesses has to be shared too. The only way out of this is casting shared away and that just looks and feels plain ugly in the code.

synchronized class ThreadsafeStack(T){
  Stack!T m_Stack;
  void push(el T){ 
    Stack!T stack = cast(Stack!T) m_Stack;
    stack.push(el);
  }
  ...
}

Also the future plan that inside shared methods memory barries will be inserted by the compiler automatically gets me concerened. Especially for performance cirtical applications this will be a performance killer and ist not even needed in most cases. (At least in my experience). As it currently stands I just try to avoid shared completely.

Here is another example. I wanted to create a shared inteface to provide cross thread debug drawing for my application:

shared interface IDebugDraw {
  void DrawBox(vec3 min, vec3 max);
}

The renderer already implemented a non threadsafe version of this method:

class Renderer {
  void DrawBox(vec3 min, vec3 max){ ... }
}

So I just wanted to implement the interface by doing some synchronization and then calling the non threadsafe version:

class Renderer : IDebugDraw {
  void DrawBox(vec3 min, vec3 max) { ... }
  void DrawBox(vec3 min, vec3 max) shared {
    synchronized(m_DebugDrawMutex){
    	Renderer self = cast(Renderer)this;
    	self.DrawBox(min,max);
    }
  }
}

This will not compile because you cant overload using shared. Choosing a different name for the function just to differ between the threadsafe and non threadsafe version seems strange to me especially when you have a feature in the type system that should exactly do that.

Another problem is that you are not able to define shared delegates. The type the compiler puts out can actually not be used in sourcecode.

class Foo {
  void bar() shared {}
}

void main(string[] args){
  Foo foo;
  static assert(0,typeof(&foo.bar).stringof);
}

If you do this the compiler will put out: “void delegate() shared” as type for bar.
If you now try to use this:

void main(string[] args){
  Foo foo;
  void delegate() shared del = &foo.bar;
}

It will not compile. Also tried all kinds of variations:

(void delegate() shared) del = &foo.bar;
shared(void delegate()) del = &foo.bar;
void shared delegate() del = &foo.bar;

None of them will work.

6.Thread Local Storage
There has been a particulary ugly bug in one of my projects connected to TLS. In total TLS was one of the most frequent causes for bugs. Most likely because I was not used to it, but still I want to mention this here:

class Foo {
  static Manager m_Manager; //unamanged class will not be scanned by GC, there is 1 manager for each thread

  this(){
    m_Manager.register(this);
  }

  ~this(){
   m_Manager.unregister(this);
  }
}

As you don’t know in which thread the destructor will be executed as the GC may run in any thread, you can not tell if the class wil actually unregister itself on the same Manager it registered itself. The next thing I tried is creating a lambda inside the destructor and send it to the correct thread to perform the deregistering, unfortunatley this is not possible because no allocations can happen inside the constructor because the GC is currently collecting.
The same problem was also a major issue when freeing OpenGL resources. As the resources have to be freed in the thread that holds the OpenGL context it would crash everytime the GC ran in a different thread than that one. I think there should be at least a warning by the compiler if TLS variables are used inside a destructor because you never know if the TLS variables actually contain what you expect.

7. Associative array invariance
This was actually the second most frequent cause for bugs in my projects. Especially because you don’t get any warnings or asserts in the debug build about it. It just goes into undefined behaviour. If possible the associative array should track all its ranges and see if one of them is still iterating. If so there should be a exception or an assert when trying to change the associative array.

8. std.concurreny TID does not support shared
This is really a pitty. The message passing mechanic prised in the TDPL book does not support shared TID types. Usually you would assume that you can share the TID structure and work on it as the message passing should be threadsafe. Casting away the shared everytime is a pain.

8. No function overloading with template parameters
This one got me especially disappointed after reading Andreis “Modern C++ Design” Book. In there he describes a trick in C++ to overload Functions with template parameters which looks in D basically like this:

struct Type2Type(T){
}

class Foo1 {
  void funcHelper(Type2Type!int p){
    writefln("a");
  }
}

class Foo2 : Foo1 {
  alias Foo1.funcHelper funcHelper;
  void funcHelper(Type2Type!float p){
    writefln("b");
  }
  
  void func(T)(){
    funcHelper(Type2Type!(T)());
  }
}

void main(string[] args){
	Foo2 foo;
	foo.func!float();
	foo.func!int();
}

Now I thought that in D it would be possible to do this without this trick as you are able to restrict tempaltes:

class Foo1 {
  void func(T)() if(is(T == int)){
    writefln("a");
  }	
}

class Foo2 : Foo1 {
  alias Foo1.func func;
  void func(T)() if(is(T == float)){
    writefln("b");
  }
}

Unfortunately this will not compile:

Error: alias test.Foo2.func conflicts with template test.Foo2.func(T) if (is(T == float))

Other then that the biggest problem with the D programming language for me is that the tools are not really major yet. But hopefully this will improve over time.

AlienFX Winamp plugin update & source

admin — Sat, 14 May 2011 14:42:00 +0000

With some help of Nils Reichert I was able to modifiy my AlienFX Winamp plugin so that it now supports the M17X R3. It should also support the M15X R3 now as the used AlienFX devices should be the same. You can find the download locations in my old post:

>>> Winamp AlienFX Plugin <<<

I also decided to release the source code for both the plugin itself and the led tester I’ve written, so that someone else can continue the developement as I don’t have time for it anymore. You can freely modifiy the programm under the following conditions:

you have to name me as original author
you may not use it for anything commercial
you have to distribute it under the same conditions then I do

AlienFX Winamp Plugin von Benjamin Thaut steht unter einer Creative Commons Namensnennung-Nicht-kommerziell-Weitergabe unter gleichen Bedingungen 3.0 Deutschland Lizenz.

Sourcecode: AlienFX-Winamp-plugin-source.zip

Light Propagation Volumes Implementation

admin — Mon, 07 Mar 2011 18:39:28 +0000

For a university project I implemented the Light Propagation Volumes technique implemented in CryEngine 3 to simulate indirect illumination. I wrote this demo from scratch using OpenGL and the D programming language.

The paper about the alogrithm can be found here:
http://www.vis.uni-stuttgart.de/~dachsbcn/download/lpv.pdf

The model and textures have been taken from the crytek site: http://www.crytek.com/cryengine/cryengine3/downloads

You can download the application for windows including the full sourcecode and documentation here:
lpv-release.zip
To fly around in the scene klick into the 3d window and press M to activate the mouse, then you can fly around using the WASD keys and the mouse. To deactivate the mouse looking press M again.
For everything else you can use the GUI.

Implementation of Light Propagation Volumes von Benjamin Thaut steht unter einer Creative Commons Namensnennung 3.0 Unported Lizenz.
Beruht auf einem Inhalt unter www.vis.uni-stuttgart.de.

D 2.0 Stacktrace

admin — Wed, 06 Oct 2010 11:48:08 +0000

I wrote a small piece of sourcecode that generates stacktraces in D 2.0 under windows. It works both with the pdb and cv debug symbol format. For Exceptions that are derived from the Error or Exception class the trace information is automatically appended, this causes all builtin D errors to get a stacktrace information. The only point where this does not work is the Access Vioaltion error, as it does not call the stacktrace callback function for some reason.

It is very easy to use, just copy the two files from the zip archive to your root source directory and import the stacktrace module inside your main file.

import stacktrace;

void main(string[] argv){
...
}

Now all Errors will get trace information.
If you need a backtrace at a certain point in your code just do the following:

module foo;

import stacktrace;

void blup(){
...
StackTrace trace = new StackTrace();
auto stack = trace.GetCallstack();
foreach(char[] s;stack)
writefln("%s",s);
...
}

Download: d2stacktrace.zip (12 kb)

It is released under the “Creative Commons by” license. If this licence causes any problems for you, feel free to contact me and we will fix the problem.

XP Users:
If you get a stacktrace with numbers only download a new version of dbghelp.dll (it comes with the windows platform SDK) and put it into your project root directory. That should fix it.

D2 Stacktrace von Benjamin Thaut steht unter einer Creative Commons Namensnennung 3.0 Unported Lizenz.

Winamp AlienFX Plugin

admin — Sat, 03 Apr 2010 12:28:00 +0000

I made plugin for Alienwares AlienFX Device that comes with every Alienware laptop and computer. Currently the following models are supported:

Area51 m15x
All Powerfull m15x
All Powerfull m17x
All Powerfull m11x

A demonstration of the plugin by Lord_Zath form NBR (www.notebookreview.com) can be found at:
www.youtube.com/watch?v=dLymX4fLVtc

Download: vis_alienfx.zip (257 kb)

Please be shure to read the README file. It contains important instructions howto setup and configure the plugin! Also I want to thank Lord_Zath for helping me test the plugin and writing the README file. A thank you also goes to all the people on NBR which are taking part in the developement of this plugin.

If you have a Alienware laptop / computer that is not yet supported and you are willing to help me developing the support for it, please contact me, and I will built in the support for your model.

Update 31.10.2010: Fixed that the plugin is crashing winamp when closing winamp or stopping the plugin. The plugin now also automatically detects on which alienware laptop it is running.

Easyhook tipps and tricks

admin — Sat, 26 Sep 2009 15:58:27 +0000

I wanted to hook the WinApi funktion CreateFile, CloseHandle, ReadFile and WriteFile to decode a device protocol. So I searched for a hooking library in the internet. Unfortunately the only good one for c++ was detour from Mircrosoft which was free only for x86 non comercial use. As my laptop runs with a x64 Windows Vista this was no option for me. So I found easyhook, a c# hooking library. It was quite easy to achive the first success with it, but then I had a few small problems for which I want to describe the solutions here.

Lets asume you want to hook a function which easy hook which executes some hooking code before and after the original call:

[UnmanagedFunctionPointer(...)]
delegate bool DSomeFunc(...);

[DllImport("somedll.dll")]
static bool SomeFunc(...);

static bool SomeFunc_Hooked(...){
  some code before original call

  bool ret = SomeFunc(...);

  some code after original call
  return ret;
}

You will notice that the code after the original call never gets executed. This is because the DllImport gives you the already hooked function. So you are calling the hooked function from the hooked function. This is prevented by the easyhook lib, by returning the flow to the calling program. (which is a very good thing, because it would lead to a infinite loop)
To avoid this behaviour you have to get the unhooked function from the easyhook library.

[UnmanagedFunctionPointer(...)]
delegate bool DSomeFunc(...);

private DSomeFunc SomeFuncPtr;

static bool SomeFunc_Hooked(...){
  some code before original call

  bool ret = SomeFuncPtr(...);

  some code after original call
  return ret;
}

And in the run method of your injected dll right after installing the hooks.

...
SomeFuncPtr = LocalHook.GetProcDelegate("somedll.dll", "SomeFunc");
...

Now your code after the original api call gets executed!

The second problem I encounterd was on hooking WriteFile:

[UnmanagedFunctionPointer(CallingConvention.StdCall,
            CharSet = CharSet.Auto,
            SetLastError = true)]
        delegate bool DWriteFile(
            IntPtr hFile,
            byte[] lpBuffer,
            uint nNumberOfBytesToWrite,
            out uint lpNumberOfBytesWritten,
            [In] IntPtr lpOverlapped);

        [DllImport("kernel32.dll",
            CharSet = CharSet.Auto,
            SetLastError = true,
            CallingConvention = CallingConvention.StdCall)]
        static extern bool WriteFile(
            IntPtr hFile,
            byte[] lpBuffer, 
            uint nNumberOfBytesToWrite, 
            out uint lpNumberOfBytesWritten, 
            [In] IntPtr lpOverlapped);

        static bool WriteFile_Hooked(
            IntPtr hFile,
            byte[] lpBuffer,
            uint nNumberOfBytesToWrite,
            out uint lpNumberOfBytesWritten,
            [In] IntPtr lpOverlapped)
        {
            for(uint i=0;i
On every WriteFile call I got a exception in the loop for i=1, because if the call comes from a unmanaged app the byte array has no size and therefore the size 1 is assumed.

To fix the problem one has to use the marshal class and change all the byte arrays to pointers.
[UnmanagedFunctionPointer(CallingConvention.StdCall,
            CharSet = CharSet.Auto,
            SetLastError = true)]
        delegate bool DWriteFile(
            IntPtr hFile,
            IntPtr lpBuffer, //changed
            uint nNumberOfBytesToWrite,
            out uint lpNumberOfBytesWritten,
            [In] IntPtr lpOverlapped);

        [DllImport("kernel32.dll",
            CharSet = CharSet.Auto,
            SetLastError = true,
            CallingConvention = CallingConvention.StdCall)]
        static extern bool WriteFile(
            IntPtr hFile,
            byte[] lpBuffer, //no need to change here because we are calling that
            uint nNumberOfBytesToWrite, 
            out uint lpNumberOfBytesWritten, 
            [In] IntPtr lpOverlapped);

        static bool WriteFile_Hooked(
            IntPtr hFile,
            IntPtr lpBuffer, //changed
            uint nNumberOfBytesToWrite,
            out uint lpNumberOfBytesWritten,
            [In] IntPtr lpOverlapped)
        {
            byte[] bytes = new byte[nNumberOfBytesToWrite];
            for (uint i = 0; i < nNumberOfBytesToWrite; i++)
                bytes[i] = Marshal.ReadByte(lpBuffer, (int)i);

            for(uint i=0;i
Now the hook works. The same works for hooking ReadFile but after calling the original function (also use tipp 1!)
For hooking I simply changed the FileMonInject example the source can be downloaded here:

C# Source