Talk:BSON File Format

RFC (kev's initial thoughts)
For third party servers, there is desire to come up with a new interchangeable world file format.

The goals of the format should be to:  ease implementation promote interoperability provide acceptable performance provide backward and forward compatibility 

Expanding on these issues:

1) Easy implementation
To ease implementation, the file format shall be one that is easily read in many languages.

2) Interoperability
As a side-effect of 1), part of the interoperability puzzle is already solved. The other pieces involve common convention of implementation and usage in file naming and internal formatting.

The format shall standardize and specify the endianness of all internal numerical datatypes.

3) Acceptable performance
The format shall provide acceptable performance. Storage of native primitive numerical types are desired to reduce complexity during both reads and writes. At minimum, this shall include:  Bytes 32-bit integers 64-bit integers Doubles 

The format shall allow advanced data types beyond these as needed.

The format shall be uncompressed by default, but a compression scheme shall be defined and common to all compliant implementations.

4) Backwards and Forward Compatibility
It shall be recognized that the file format must evolve to new game features. Therefore, the format shall be internally versioned so as to prevent mismatched file vs. parser usage.

When possible, newer readers should be able to read older files (backwards compatibility). Common utilities should be provided to ease upgrades when incompatible changes are needed, and the file format should be flexible enough to allow enhancements that were not known during previous specifications (forward compatibility).

More Random thoughts
A quick survey shows that http://bsonspec.org/ meets many of these requirements. With some light conventions on top, it is one option. I will do some more research and see if there is anything more suitable.

The official utilities should probably be written in a widely used language such as python. These will include NBT<->MCBSON, a test suite, and standard utilities.

Deoxxa's IRC rant
i strongly suggest not using sqlite honestly if not for the potential performance issues then for the added complexity at least i think a fairly quick and flexible system could be implemented with 3 files, maybe 4 one file for block, light and meta data one for entity data (structured) and one that acts as an index mapping x/z chunk coordinates to offsets in said files the first file would never need to be completely rewritten, as that part of the chunk data is all the same length you could read it in and cast it to your application's data structures the second one would occasionally need to be rewritten (in the event that a chunk's entity count grows) and the third would never need to be completely rewritten either since you could store the x, z and offset values in even 64 bit integers and it would probably never get too big to handle