5:32 p.m.
Post via forum by datahead <cjj_009@xxxxxxxxxx>:
Grimshaw|lap, a friend of mine from the SFML chat room suggested that we use XML for our
level format only during development and that binary (representing the in game objects) be
used for production purposes.
Thus, for campaign levels, we would use our usual XML format if developing and if we need
to debug or research trouble points. We would then convert it all to binary such that we
can mass load data to construct C++ objects at runtime for the final version.
For level editing for users, it would produce XML that they are able to see. The first
time they run the level, it would run slower, as it caches this into binary. It would
then load this binary every time they play, unless they make changes again, in which case
they would take a performance hit the first time it deals with the XML again.
He pointed out this is a standard practice in many production game companies and that he
would expect out level loading to be slow without this.
--
Sent by Chessboard.
10:08 p.m.
datahead <automailer(a)forum.secretchronicles.de> writes:
He pointed out this is a standard practice in many production game
companies and that he would expect out level loading to be slow
without this.
libxml2 loads XML faster than an average binary file loader can load a
binary file, because it is a highly optimised library with years of XML
parsing efforts that went into it. So unless you spend a lot of time in
creating a sophisticated binary file format parser, you’re going to have
a hard time in beating libxml2’s performance, especially as we use the
SAX parser and not the DOM-based one (i.e. the XML is traversed exactly
once).
Apart from that I find binary file formats a nightmare for debugging
purposes. Without really good reasons I don’t think this is a good idea.
Vale,
Quintus
--
#!/sbin/quintus
Blog: http://www.guelkerdev.de
GnuPG key: F1D8799FBCC8BC4F
12:35 p.m.
Post via forum by datahead <cjj_009@xxxxxxxxxx>:
This was grimshaw's response:
[quote]
Yeah, reading C++ objects in the binary form will never be 100% efficient, but it can
approach that pretty close. But it will depend on how you design it too.
Even with some imperfections it will beat the hell out of XML, as it needs a lot of
lexical parsing and making sense of all that data into runtime structures, which then need
to be iterated and the data extracted for the final in memory structs.
If you are able to design your scene objects around POD structs, then you can optimize
this to a really fast system. You'd have to commit to POD data fields only, but then
you could achieve an almost zero overhead level format.
For example, I inspected your level format, and there is a lot of <sprite> objects.
In C++ it could look like struct SpriteInstance { char texture[256]; float x; float y; }.
This is just an example that doesn't try to cover the real scenario.
There, you abdicate a std::string utility for a flat data layout. When you do this, you
can indeed guarantee a 2 step loading of ALL the sprites in your scene:
{
std::vector<SpriteInstance> sprites(numSpritesInTheFile);
fread(&sprites[0], sizeof(SpriteInstance), numSpritesInTheFile, filePointer);
}
You can't go much faster than this for data file loading. In theory sure, you could
compress or quantize or whatever to try to squeeze some more seconds out of it, but its
probably not worth.
[/quote]
He estimated binary as being 150% faster than the xml implementation, though he emphasized
that estimate is very rough based on intuition and experience only.
I'd be interested in performing a benchmark test to compare the two methods, though it
will probably be some time before I find time to do this, given the need for SFML port
work and story reviewing for TSC.
[quote]Apart from that I find binary file formats a nightmare for debugging
purposes.[/quote]
We would be using XML for most debugging, and we would make this the most visible format
to end users who create levels. As for the binary used for the campaign release or that
is cached for user levels, we would keep it as simple as possible using the POD approach
that Grimshaw suggested.
We might end up doing things slightly differently if it suits us, but I did get something
somewhat similar to this with POD objects to work in order to "auto serialize"
objects in .NET for a space shooter game once.
--
Sent by Chessboard.
2:37 p.m.
datahead <automailer(a)forum.secretchronicles.de> writes:
Post via forum by datahead <cjj_009@xxxxxxxxxx>:
This was grimshaw's response:
[quote]
Yeah, reading C++ objects in the binary form will never be 100%
efficient, but it can approach that pretty close. But it will depend
on how you design it too.
I have to agree that I didn’t understood the point first. With “binary
format” I thought you wanted to use a structured binary file format with
a specification other programs can implement easily. That is, like XML,
but with raw bytes, like a binary protocol that is spoken over a TCP
line. I didn’t realise you meant a memory dump.
Which brings me to my first point. Sure, a memory dump is fairly easy to
generate and reload. However, it lacks any kind of specification and
makes it near to impossible for other programs to read it in. This was
for example the problem with MS Word 2003 .doc files; they were
basically memory dumps of MS Word, which made it so hard to create an
implementation for them in OpenOffice. This is, as I understood it, the
main reason why MS Word switched to — guess what — an XML-based format
(OOXML). There was an attempt to write a level generator on the old SMC
forums; things like that will become overly complicated with memory
dumps from TSC.
For example, I inspected your level format
Regarding the current level format, we are not content with it
anyway. It should not be used for comparison. There are even two
tickets on the tracker concerning themselves with the level format:
* https://github.com/Secretchronicles/TSC/issues/169
* https://github.com/Secretchronicles/TSC/issues/354
And one for the savegame format:
* https://github.com/Secretchronicles/TSC/issues/380
{
std::vector<SpriteInstance> sprites(numSpritesInTheFile);
fread(&sprites[0], sizeof(SpriteInstance), numSpritesInTheFile, filePointer);
}
So, here’s the memory dump (or rather the loading of it). This results
in a file format depending on the memory layout of the target
machine. How does one distribute a level then? Say, if I want to run my
level created on a little-endian system on a big-endian one? What do we
do when we expand the sprite struct with new members in later versions,
which will break any existing memory dump? What happens if the target
system uses another implementation of the C++ standard library, which
lays memory out differently? Oh okay, that’s why you used a C struct
instead of a proper object-oriented C++ class, they have more defined
memory behaviour. So sacrifice OOP/code design for speed. Since I
usually prefer a well-layed-out and -designed codebase over speed, I find
this an unusual approach.
Apart from that I suspect such code to rely on undefined behaviour. Is
there a gurantee in the C++ standard that the elements in an std::vector
are really consecutive in the underlying memory, with nothing
in-between? This seems like an implementation detail one should not rely
on (and doing so violates the secrecy principle anyway). I could imagine
the container class to encapsulate the respective items in an own
element class instance, thus breaking the assumption of consecutive
memory. If it indeed causes undefined behaviour, we would make ourselves
dependant on a specific compiler/libstdc++ version, which is something
definitely to avoid.
[quote]Apart from that I find binary file formats a nightmare for
debugging purposes.[/quote]
We would be using XML for most debugging, and we would make this the
most visible format to end users who create levels.
So instead of one level loader to maintain we then have two level
loaders to maintain, which may expose different behaviour due to bugs
that will happen anyway. I don’t find this a good idea.
Vale,
Quintus
--
#!/sbin/quintus
Blog: http://www.guelkerdev.de
GnuPG key: F1D8799FBCC8BC4F
1714
days inactive
1718
days old
tsc-devel@lists.secretchronicles.org
3 comments
2 participants
participants (2)
-
datahead -
Quintus