ImageMapDatablock... why a datablock?

#1 IMHO, it's more because resource management should be left up to the engine rather than the programmer. I'm a big fan of not worrying about the life cycle of dynamic objects. I like garbage collection. I just want to get in there, toss them around, and forget about the details.

#2 Jason,

Nothing is really stopping you from doing this now or in v1.0.2.

$im1 = new t2dImageMap("helicopters.png");
$im2 = new t2dImageMap("airplanes.png");
 
$ss1 = new t2dStaticSprite() { sceneGraph = MyScene; };
$ss1.setImageMap($im1);
...
$ss1.setImageMap($im2);
$im1.delete();

To be honest, you wouldn't be able to create an image-map as "simple" as you mention here as you'd need to specify all the other parameters you needed to define the frames which, in reality, makes your code as ugly as hell. Also, this syntax makes it look like it's good practice creating lots of identical image-maps and assuming that T2D just deals with it but this wouldn't be the case.

On-top of this, across the network, these objects don't have any distinction from other objects. Datablock-transfer is part of a pre-stage to connecting to a server. The server downloads these datablock resources to clients before the objects are created. On the server, they don't actually load anything whereas when downloaded to the client, they do. This is a future reserve though.

Having a slightly different syntax sets it apart from this trivial create/destroy view but there's nothing stopping you from doing so. Create a datablock as normal, use it, then destroy it. Be aware though that it's generally a bad idea to do so though.

There's work going on to be able to specify how/when the loading/unloading of image-data is controlled, not explicitly but with options that allow you to do things like tell an image-map to load/unload the images only when they're assigned/not-assigned to an object (which can be slow and why we've avoided it so far). Also, the ability to cache the calculated packed image-frames to disk as well as other stuff.

As soon as we do this, we'll be supporting people that say, "t2d is really slow". The reason will be because they're creating/destroying an object and inbetween each one, the T2D resource manager will be loading/unloading an image-map. Getting across the impact of some of this power will be a challenge in itself.

Essentially, the work going on isn't to make them dynamic, they already are, but to provide control for the dynamics of them.

In the end, a datablock is just another object so if you delete it, the memory used disappears as expected but you'd normally do this at a "level end" point where you know that all objects referencing it are removed.

EDIT: Just thought I'd add that I totally understand that having a "t2dImageMap" would make it just another object in T2D that's created/destroyed in the same way but this is exactly the view we've tried to avoid. Shared, static resources have always been put into datablocks in TGE. If the powers that be said move it to its own object to make the syntax better, I'd do iy.

- Melv.

#3 Hey Melv,

First thanks for your perspective. I'm not looking for a change, just understanding. So you've said something that surprises me, which is I can already create and destroy datablocks now, even though it's not supported. Really? That would be awesome if it's true. Can I get an example (from anyone) for how to:

1. dynamically create a datablock
2. delete a datablock
3. Can I put datablocks in a SimGroup/Set?

for #1, I'd really like something where it was of the form:

$images[0] = "helicopters.png";
$images[1] = "airplanes.png";
$images[2] = "tanks.png";
$images[3] = "boats.png";
$imageCount = 4;
 
for (%i=0; %i<$imageCount; %i++)
{
    datablock t2dImageMap($imdb[%i]) { imageName = "~/client/images/" @ $images[%i]; mode = "cell"; ... };
}

Can I do this? Better yet, can I stick the datablock in a SimGroup and then nuke the SimGroup at level end?

#4 Hi there, I wanted to do this too, so I added a
function to load the datablocks to my class
I didn't know if this was safe or not, but it seems
to work, basically:

function map001::loadDatablocks (%this)
{
	datablock t2dImageMapDatablock(cieloexImageMap)
	{
		imageMode = CELL;
		cellWidth = 128;
		cellHeight = 128;
		imageName = "~/client/levels/001/cieloex";
		filterPad = true;
	};
	updateLoadingPercent (10);
	...
}

new ScriptObject (clsMap001)
{
	superclass = mapLevel;
	class = map001;
	levelMap = "~/client/maps/lvl001.map";
};

function map001::create ()
{
	%map = new ScriptObject (: clsMap001) { };
	// load graphic datablocks...
	%map.loadDatablocks ();
	...
	return (%map);
}

function map001::free (%this)
{
	%this.tileMap.deleteAllLayers ();
	%this.tileMap.safeDelete ();

//	// delete graphic datablocks 
//	cieloexImageMap.delete (); //<-- THIS CRASHES
	%this.delete ();
}

As you see is possible to show a discrete progress bar
because if there are a lot of data there is going to be a little
delay in loading the graphs...
But I really wanted to ask Melv, Why I can't delete
inmediately the datablocks? this is the reason the code in
cieloexImageMap.delete () is commented...
because sometimes T2D crashes, Is there a safeDlete()
for datablocks too? I couldn't find...

#5 @Adam: The only reason I can think of is that you've still got objects that are dependant upon the imageMap and you've just pulled it out from under its feet. This'd be even more disasterous if you did it in a callback from that object. I can see tile-map stuff in your code so it looks like you could have a whole set of tiles that are using it.

This is why we keep saying, datablocks are not "meant" to be deleted. What we're really saying is that you have to be damn careful. There have been changes to the alpha, and I know of a few places that have been missed, where the reference to a datablock uses the safer SimObjectPtr<> class which reference-counts the objects. As I say though, some objects don't use it yet but it's on the list of things to do.

Even when we stop it crashing when you pull the resource from underneath the object, you've still got an object not behaving well as it's got no resource. Again, bad, bad, bad.

The extra work I mention is so that you can use auto-reference counting so you're not explicitly creating/deleting things; you just define what you want and it comes into play when you use it or, if you've organised your objects scope well, you can create/destroy them as you need them, say at level load/end etc.

There's no safeDelete for datablocks as they're not T2D objects, they're a core-engine object. Safe-delete was something I conjured-up to allow people to delete T2D objects in script-callbacks without worrying about references.

Anyway, datablocks won't change from the format they're in, that's a promise, but we will be adding extra options that allow you to control them in whatever fashion you want.

Back to the crashing though; if you get a crash like that, have a look at the debug and report where it is. More than likely it'll be in a render function somewhere. I'll definately fix those up as soon as they get reported.

- Melv.

#6 So, how does memory management work?

Let's say I have a large RPG world. It has a number of towns, dungeons, etc. The main overworld map itself has a gigantic tileset, to allow for good visual diversity in the tilemap. Each town, dungeon, etc, has its own unique tileset, to keep the different towns and such from looking similar to one another.

If a player goes into a town, the main map's imagemap datablock needs to go away. Not just become unused; it needs to be physically removed from memory. This will allow sufficient room for the town's imagemap(s) to come into being. When the player leaves the town, that town's imagemap(s) go away, thus providing room for the main map.

I can't say that I fully understand everything that you have explained on this subject, but as I understand it, datablocks aren't meant to go away ever. So, in my above example, going from town to town and dungeon to dungeon would cause the game to eat up an ever increasing quantity of memory, to the eventual point where the graphics card gives up (out of texture space) and/or the game starts trashing the harddrive frequently when leaving an area.

Is that correct?

#7 Not at all. It has been said that it's a bad idea to destroy them because there hasn't been the support, so far, to deal with them being created/destroyed on the fly. This is changing though and very soon! There's a slight mix-up between people talking about datablocks in general and image-maps. Datablocks were originally designed to define a set of static data on a server but the server doesn't load the resources such as 3D shapes, imagery etc. Then a client connects and the server transmits these datablock definitions to the client which runs them and does load the resources. All the definitions are stored on the server. This is part of the networking framework and is used by the underlying engine. T2D uses the same construct, the datablock, to define images as they are again, static resources.

With this in mind, datablocks were not intended to be created/destroyed on the fly because this would mean transmitting this change to all the clients and blah blah blah. In T2D though, this "don't delete" hangover has been carried forward a little, perhaps too much. Discussions over whether you should/shouldn't delete "datablocks" have been synonymous with image-maps and although I'm guilty of saying it, it's not strictly true, certainly outside of a networked environment. In the end, a datablock, is just a standard TGE object and can be used just like any other even if it does have a different creation syntax.

In my brief statement above, I've not given justice to the work that I'm into now but I've been over this so many times that I tend to just fly over it, sorry for that.

So far, we've seen a large range of people using T2D from people just starting out who, quite frankly, don't want to be bothered by resource-management and probably are not going to be releasing a game anytime soon but want to just get on with learning game logic in general. Then we've got the people who, quite rightly, want to keep a tight control over what's being allocated and who want different ways, according to their game-type, to deal with this. We've also got the people in the middle who understand the need for resource-management and are starting to get to grips with it. Okay, enough of putting people in boxes!

We don't want to dictate anything here but want to offer a set of options, some explicit, some implicit. The next release for image-maps will simply add a few extra field-options so there's no compatibility issues with alpha#1.

The next image-map release should allow you to...

- Allow image-maps to be generated at any point.
- Allow image-maps to be deleted (at any point) and any objects referencing them will keep working but remove any image-map association and therefore not crash!!

The two above allow people who want explicit control over memory management even if under some circumstances this isn't neccesarily always a good idea.

Then allow image-maps to load/unload resources (datablocks are still around but their resources disappear) in a couple of different ways...

- Explicit load/unload control from the image-map via a set of functions.
- Field-option to tell image-map to dynamically load/unload as resources reference it.
- Field-option to allow the image-map to destroy any object that is referenced to it automatically when itself is deleted.

... continued ...

#8 From the above list, you would be able to configure image-maps in different ways, dependant upon how you want to use them. There'll also be a global default that you can change so you're not forced to specify a choice in each and every image-map but use a standard way for resource handling.

In your example above, I guess you could take control over the image-maps yourself and unload/destroy one and load/create an alternative. Or, if you just wanted to point your tileset to a new image-map (and you'd configured the image-map to unload when it hasn't any references) then you could do this and the old one would unload and release its resources.

If you've got the notion of a level with lots of image-maps and dependant objects all active; upon level-end, you could simply iterate a SimSet which contains a list of the level resources and destroy them and everything would go. The good thing here will be that you won't care on the order of deletion if you've got a mix of image-maps, script-objects, T2D objects in the list.

There are obvious downsides to offering modes such as auto load/unload with regards to references which would be people starting-out wondering why the image-map is thrashing loading/unloading as you create/destroy a sprite that references it but that's a case for good documentation on how/when it's best to use a specific mode.

A common mode would be to just explicitly create/destroy all level-objects as and when you need. As we work through this alpha, there shouldn't be any crashes experience with destroy underlying image-maps associated with objects.

I just thought I'd also add here that the new image-maps are being generated in two stages, the first was the new image-packing, the second was the dynamic load/unload/referencing stuff. The alpha#1 got the first part only.

Okay, it's late here and hopefully I've explained it enough. Probably a little rambling as I'm tired as hell.

- Melv.

#9 Thanks for the detailed description Melv, it looks like good stuff, I can't wait.

In an effort to gain more control over memory usage in our project which uses 1.0.2 we cloned the deleteDataBlocks console method (simBase.cc) and extended it to scan RootGroup instead of just DataBlockGroup, and made it check for a new variable we added called 'transient'. If transient is true, it deletes the resource.

This is very useful for temporary (e.g. level/world/stage based) objects, you just put 'transient = true' into the applicable datablock/simGroup/whatever and whenever you call 'deleteTransientObjects' it scans through and quickly deletes them, immediately freeing up related memory. It is however pretty dangerous in 1.0.2 because the responsibility is entirely up to us to make sure there's no hanging references out there, which we've already tripped over a few times before getting it nice and safe.

It sounds like the safety checks coming for image map resources would make this much better!