Methods of Packing Animation Data

To make things simple, I think I can use a fixed struct for the animation data. To put it simply:

typedef struct DashKeyFrame {
int frame;
int bone_id;
int parent_bone_id;
char use_position;
char use_rotation;
char use_scale;
char nop;
vec3 position;
vec3 rotation;
vec3 scale;
}

I think the above struct will allow me to use a fixed struct for each keyframe. Since for every frame, and every bone we need to have a transformation matrix to tell the mesh how to deform. But with the use of interpolation it means there are a lot of values that get automatically get filled in. And we can’t make the complete assumptions about just filling in all of the keyframes, otherwise the data that gets saved into the file and read from the file becomes drastically different. Also in a lot of cases the use of nested arrays is used to manage this information, and even though it’s redundant, just including the frame number, bone id and bone parent id in each keyframe removes the need to use any kind of nested structure in the data.

As for the character boolean values, we have one value that tells the importer to use position or not, rotation or not and scale or not. The effect of this is that there can be alot of data with default zero values are included in the data, but in this case simplicity and a fixed structure take priority, so I won’t lose much sleep if a few extra bytes is included here or there. The one weird part about all of this is the frames per second value, which I figured I might as well include as the fourth byte, otherwise it’s just a no operation byte anyways. So while I’m okay with wasting bytes for the sake of having a fixed struct, we also might as well use bytes that aren’t really being used for anything. I think this makes a change to the animation entry.

typedef struct DashAnimationEntry {
char animation_name[0x20];
unsigned int animation_id;
unsigned int offset;
unsigned int offset;
unsigned int number_of_keyframes;
unsigned int frames_per_second;
unsigned int number_of_frames;
unsigned int nop[2];
}

So I end up adding two nop values to the end of the animation entry, and then one nop into the keyframe values. But I guess that’s acceptable.

Alternative Media

After using my last post to think through my blunder of buying the geekwork case, it gets even worse, but I’ll get to that in a second. For my previous post I came to the conclusion that a normal Pi 3B+ hosting an access point is the correct way to go. And after creating the pi with the access point, I can see that if I need any extra space for Next Cloud or otherwise, I can either buy usb flash memory or a usb mSata drive. What makes all of this worse is that in the time I was waiting for my hard drive and geekworm case to be shipped, the prices of storage really fell out the bottom. In this short amount of time, several prices have been sliced to half making the trade offs of price versus space I was running into before a lot more one sided in the favor of flash storage. That being said with a 128GB usb stick now coming in at around thirty dollars, it makes it a lot easier to either pick up a stick, or maybe even hold out until there’s an even better deal down the line.

Which kind of leaves what to do with the alluminum case and hard disk I bought. After after thinking about it, I think the best use of the case and hard drive is probably to host a network storage drive up stream from my pi access point. This means that I should be able to get wired transfer speeds over the local network and then a pretty dedicated connection for the few clients I have connected to the access point. For a host I think I’ll use my tinker board, which kind of kills two birds with one stone, as the Tinkerboard has two gigabytes of memory and gigabit Ethernet, but it’s been sitting in my drawer for the last year, because it’s non-standard. So I can feel better about using all of the non-standard “junk” i have laying around to use the barrel jack to power the tinkerboard, to host the two terabyte drive on my network or act as a backup.

Piboy Pocket

Quick outline for the “Piboy Pocket”. Originally the idea was to use either a Raspberry Pi zero, or the Raspberry Pi computer module with an e-ink screen to replicate a Gameboy Pocket. Though after finding out how slow the refresh rate on an e-ink device screen is, that doesn’t seem like a viable option, at least for now. And if anything I think the Arduboy fits this niche pretty nicely already. And rather than a Pi the Arduino makes more sense for this task. Not to mention that $50 for a completed product than trying to roll my own (in this case). That being said it looks like the arduboy is out of stock, so it might be worth ordering one if I ever come across one.

Dash Model Format Version 2

I think I have the header defined for the next version of the dash model format

Offset 0x00 0x04 0x08 0x0c
0x0000 DASH 2 offset length
0x0010 IMG num offset length
0x0020 TEX num offset length
0x0030 MAT num offset length
0x0040 VERT num offset length
0x0050 FACE num offset length
0x0060 BONE num offset length
0x0070 ANIM num offset length

IMG

Images and animations will be in archive. For image, i think the archive struct will be:

typedef struct DashImage {
	char image_name[0x20];
	unsigned int image_id;
	unsigned int offset;
	unsigned int length;
	char image_extension[0x04];
}

TEX

For textures, since we don’t need much I think we can use the following:

typedef struct DashTexture {
	char texture_name[0x20];
	unsigned int texture_id;
	unsigned int image;
	unsigned int wrapS;
	unsigned int wrapT;
	unsigned int flipY;
	float rotation;
	vec2 center;
}

Generally which image to use, how to wrap S and T, whether to flip Y or not, the rotation (in radians) and the center. Probably won’t use all of these, but it should allow for these values to be utilized if needed, even if I never use them.

MAT

For materials, the main issue I think is how to manage blending.

typedef DashMaterial {
	char material_name[0x20];
	unsigned int material_id;
	int texture;
	float alpha_test;
	float opacity;
	char vertex_colors;
	char blending;
	char blend_src;
	char blend_dst;
	char side;
	char transparent;
	char visible;
	char skinning;
	DashColor diffuse;
	DashColor ambient;
	DashColor specular;
}

For materials, generally the aim is to go for something between lambert and phong. We only need one texture coordinate, since bump, environment and lighting maps are beyond the scope of what the goals of this file format are. Values that are not defined can be ignored, like blending should be 0 is no blending is defined, ambient and specular can have 0 for the fourth bit to indicate not to use them. And as for vertex colors, if the face defines them, then i’ll set the value for the material if needed in the importer, but I don’t think the value is needed in the struct, though I might add it just the be safe.

VERT

Vertices should be pretty easy.

typedef DashVertex {
	vec3 position;
	unsigned short indices[4];
	vec4 weight;
}

A vertex is defined as a position combined with a weight and index. If there is no weight (or index), then just fill in zero.

FACE

The face we already defined in a previous post. I don’t really see any need to change anything. The only part that annoys me is the use of nop to fill in space between four byte boundaries. Though I could just make the indices unsigned ints, but that would be kind of stupid when passing to the graphics buffer. So I might take another look and move some crap around.

typedef DashFace {
	char materialIndex;
	char useVertexColors;
	char useVertexNormals;
	char useTexCoords;
	unsigned short indices[3];
	unsigned short nop;
	vec4 vertex_colors[3];
	vec3 vertex_normals[3];
	vec2 tex_coords[3];
}

BONE

For the bones I kind of want to throw everything in (including the kitchen sink). Calculating values for bones can be annoying, so if the program provides these values it would probably be best to use them, or at the very least be required to know how to calculate them, so they can be verified.

typedef DashBone {
	char bone_name[0x20];
	int parent_bone_id;
	int bone_id;
	vec3 position;
	vec3 rotation;
	vec3 scale;
	mat4 matrix;
	mat4 inverse_bind_matrix;
}

So pretty much include everything. Local position, local rotation, local scale, local matrix, and inverse bind matrix. If an importer works better with one more than the others it can just be used, otherwise, these values can be calculated as needed.

ANIM

typedef DashAnimationEntry {
	char animation_name[0x20];
	unsigned int animation_id;
	unsigned int offset;
	unsigned int length;
	unsigned int frames_per_second;
}

Animation entry is just a carbon copy of image entry, since it’s basically an internal archive. For the actual animation data, there are three options, fcurves, so a value for a given axis for a given time, or a collection of key frames for specific frames, or lastly a key frame for every bone, for every frame in the animation. I’m really tempted to go with the last one because it makes reading and writing stupidly easy, and these values can be optimized by the importer if needed.

DashGL Format, Support and Faces

In general I’m pretty happy with the texture and material approach I came up with in one of my earlier posts. By standardizing the structs with respect to textures and materials and keeping the options limited, we can simply our header file. Though this leaves us with a few issues. If it’s possible to simply most of our struct type entries, is there a way to standardize images, faces and animations so that these only have one entry? And then in terms of entries, is it possible force a fixed header, or will the header depend on the type of the content?

In general the goal is to create a binary file format that only supports a fixed number of attributes and is geared towards models that were around in the mid-1990’s. So basically meshes, rigged meshes, and rigged meshes with animation. In terms of faces, we’re looking to normals, vertex colors, and texture cords and whether to use them or not. Which means we could potentially use a face struct similar to the following:

typedef struct Face {
char materialIndex;
char useVertexColors;
char useVertexNormals;
char useTexCoords;
unsigned short a, b, c, nop;
vec4 a, b, c;
vec3 a, b, c;
vec2 a, b, c
}

So that basically for each face we have a material index, whether the face uses vertex colors, vertex normals, and/texture coordinates. For materials I imagine it would be difficult to use more than 256 materials in a single file, but if that ever becomes an issue we can just increment the file type number. For the boolean values it would probably be simple to just use bit flags, but I think that would be cramming things down too much. Using structs means that we can include the definition of what each value is doing directly into the source code, where as with flags it would have to be describe somewhere else.

Following that we would have the indices for the a, b, and c values for the face. For the indices I think that we would define a vertex as a x,y,z position combined with a weight. So I might force the vertex list to be declared with the weights required, and then just set to all zero if not used. It seems kind of wasteful for files that might not use it, but in comparison I think the saving from using binary over text are going to be sufficient. And it also has the benefit that there are no major structural changes in the file for if a mesh has weights or not (i’m looking at you .dae). Which means we could get the file header down to something that looks like:

IMG
TEX
MAT
VERT
FACE
BONE
ANIM

And then that would make it easier to just simply require all of the things. For IMG and animation, I think we could declare the number in the header, and then just have the number of bytes before each image is declared and then just read the number of images from there. And then do something similar with animations. What would probably be more “computer-sciency” and cleaner would be to have a look up table at the top of the IMG and ANIM sections to be able to have a global offset to just to to read the information. So the section would basically have it’s own mini-header. That way someone could read the header, and then read the offset to a given image rather than having to loop through all of them. It would also make the file a little more forgiving to edit (not that I can see too many people editing by hand), but it has the added benefit of having more nerd points.

But packing all of the attributes in this way means that I could have a very simple header (type, offset, count) with a fixed number of entries. And then a simple list to determine what kind of mesh the file is. If there are bones, then it’s rigged, no bones, then it’s a mesh. Bones and Animation, then it’s rigged with animation, though the main detail is to look at the bones, to determine yes or no.

GeekWork Case

I think this experiment might go into the “failed” bin. At home I’ve been increasingly using Raspberry Pi’s to replace any computing needs I might need on a network. Why raspberry Pi’s? Because their ubiquitous, it’s easy to find documentation for them, they only require a micro-usb to power them, generally that amount of computing power is more than enough for a web server, and most importantly it’s a good learning experience to create your own devices and match them together rather than buying something complete.

Specifically my goal for this case is to have an access point for my house. This goes hand in hand with my Android to Lineage experiments. But basically I want to have LineageOS on my cell phone. Then I want to host NextCloud at home. And then to use NextCloud on my cellphone, my NextCloud instance needs a valid https certificate. Which means that my plan was to use my super-cheap cloud server to verify my certificate, have a Raspberry Pi act as an access point, and then use nGinx to forward the traffic back to my NextCloud instance.

Though as I’m writing this I’m realizing a pretty big mistake that I was making. My intention was to make a single device that would perform all of these functions. But In reality I think it would make a lot more sense to simply have a barebones access-point with minimal functionality on it, (just the access point and a DNS server), and then start adding Raspberry Pi’s into the network with fixed ip’s and DNS entries and just have them all forwarded from the central access-point-pi.

Planning out the next Dash Model Format

In the last post I examined gltf and I think it has too many issues to get involved with. That being said, I definitely won’t complain if there is some ground swell of support and devs can magically make the file format viable, but until then I’m not holding my breath. I think it would be a better use of my time to address the issues with the Dash Model Format and amend them so that I can start working on a Noesis plugin or something.

One thing that became clear after looking back over the gltf definition is that it does make sense to split the image and the texture, which is something I was having trouble with. As the attributes for textures can be a set struct, but the image is a variable length file. Granted I can have the image merged into the texture, but it makes the file format really ugly. So what I’m thinking of now, is having the images listed individually, with the count being the number of bytes. And then textures and materials can be structs with the count representing the number of structs in the array.

Type Offset Count

IMG 0x100 0x123
IMG 0x130 0x395
TEX 0x400 0x2
MAT 0x420 0x2

Which I think makes the image, texture and material management really clean looking. The only other consideration is the option to add names into the file. Something I’m tempted to leave out completely. But if I do put it in, I can allocate 0x20 bytes to a name string and the beginning of the texture and material structs respectively to define a name. Since Images are already using the byte length for the count, I don’t think it’s a good idea to add in names, so if it’s needed it can be included in the texture if needed.

My Issues with GLTF

I have more than my fair share of issues with the .dae or Collada file type. Generally the fact that it’s an XML file format makes it a non-starter and generally very difficult to work with. The use of tags and id’s makes ti very messy to export to as a file format. And even if you try to use json as a base format to work with, you’ll never really know what you’ll end up with until you actually try exporting the file, which generally means you’re going to spend most of your time messing with strings. But in addition to that, the example models don’t provide enough examples, and there are functionalities, such as the ability to include textures as base64 images, that are just not supported.

In general it looks like .dae came out, and a lot of people at the time put a lot of effort in to make it work, but no one has supported the file format beyond that because it sucks.

So when .gltf was announced as a json format, I was cautiously optimistic that there could be an accessible standard format for 3d models. After looking at the file spec, my optimism quickly faded. Not that I’ll complain if the file format gains wide spread adoption and is supported by a wide number of applications to the point where I don’t have to worry about it. But there are several aspects of the file definition that I find baffling and may serve as a road-block to anyone supporting it.

Scenes

    "scene": 0,
    "scenes": [
        {
            "nodes": [
                0
            ]
        }
],

Not a huge deal. In general scenes are hard to get a grasp on when you’re starting to build tools by exporting cubes. I’m not sure I’ll need to use scenes anytime soon, but I can understand why they are there. What confuses me about the scenes though is the way they are implemented. If you go the other way, you can make a scene with a list of models and lights. Though I guess that’s what’s going on here, you define a list of objects or “nodes” and then just define by index, which objects or “nodes” are in any given scene.

Everything is a Node

    "nodes": [
        {
            "children": [
                4,
                1
            ],
            "matrix": [
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                0.0,
                -1.0,
                0.0,
                0.0,
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0
            ]
        },
        {
            "mesh": 0,
            "skin": 0
        },
        {
            "children": [
                3
            ],
            "translation": [
                0.0,
                -3.156060017772689e-7,
                -4.1803297996521
            ],
            "rotation": [
                -0.7047404050827026,
                -0.0,
                -0.0,
                -0.7094652056694031
            ],
            "scale": [
                1.0,
                0.9999998807907105,
                0.9999998807907105
            ]
},

This is one that I find confusing but everything is a node. Models are nodes, bones are nodes, lights are lights. I can kind of understand as in the context of a 3d library, the actual content of any given 3d object doesn’t really matter. From the library everything is just a transformation matrix that gets pushed onto the matrix stack. In the case of a list of bones, I can probably make the root bone the first “node” followed by the list of bones following that. And then have the mesh and skin declared at the end. The mesh that gets declared at the end is probably going to be the index that gets declared in the scene array.

Though this works for one model as you get a close to being internal with the bone numbering. I would be scared to have more than one skinned mesh in a single gltf file, and if that would require the exporter to offset the bone list of a given character by the index in the array. So if I have a character, a weapon and a pet. I could see how you could add the weapon as a node in the characters hand. But a pet that has it’s own skeleton and animations, it would make a lot more sense to just declare a new gltf file. Which is also kind of annoying as there’s no difference between individual assets or assets included as a scene. A gltf file will generally have to be seen as the root of a given scene by an importer. Which is why it would kind of be nice is the specification had the option to leave out the scene definition entirely to specify a single asset.

Meshes are Stupid

    "meshes": [
        {
            "primitives": [
                {
                    "attributes": {
                        "JOINTS_0": 1,
                        "NORMAL": 2,
                        "POSITION": 3,
                        "WEIGHTS_0": 4
                    },
                    "indices": 0,
                    "mode": 4,
                    "material": 0
                }
            ],
            "name": "Cylinder"
        }
],

I can understand the way meshes are declared. They basically have meshes divided up into draw calls. So if I have a character with a body and a face with two materials, then I’ll have to declare two primitives. Not bad right? Well it doesn’t really look that simple. The way I try to declare models is with one list of unique vertices, and a list of materials. And then primitives can just be a material number with a list of face indices. In this case the position, normals, joints and weights are all included in the primitive. Does this mean I need to include the same global list of vertices twice? Or does it mean I need to break up the part of the mesh so that each part is effectively it’s only model? So that in this example the body is vertices 0-900 and then the head is 0-100, rather than just being one long list of vertices for the geometry, and then split into two draw calls for the faces. I’m not really sure, it’s confusing.

Skins, the Confusion Continues

    "skins": [
        {
            "inverseBindMatrices": 13,
            "skeleton": 2,
            "joints": [
                2,
                3
            ],
            "name": "Armature"
        }
],

Aside from “joints” I think I’m familiar with what’s going on here. The inverseBindMatrices is a list of matrices with the inverse transpose of the parent bone so that rotations from the animation can be calculated relative to the child bone for the t-pose. The skeleton is the list of bones. But I’m never really sure what “joints” mean, or where these indices are referring too. This was a similar problem to .dae where you had to manage a massive number of cross referenced id’s and names that should have been implicitly understood within the context of the file anyways. GLTF mostly solves this problem with indices, but the problem is they have it broken down into so many components that it becomes a problem of managing what each index is actually pointing to.

Accessors and Buffer Views are Stupid

    "accessors": [
        {
            "bufferView": 0,
            "byteOffset": 0,
            "componentType": 5123,
            "count": 564,
            "max": [
                95
            ],
            "min": [
                0
            ],
            "type": "SCALAR"
        },
        {
            "bufferView": 1,
            "byteOffset": 0,
            "componentType": 5123,
            "count": 96,
            "max": [
                1,
                1,
                0,
                0
            ],
            "min": [
                0,
                0,
                0,
                0
            ],
            "type": "VEC4"
},

In general the way gltf manages buffers is actually pretty good. I wish there were only two ways to manage buffers, 1. either as base64 string, or 2. as a binary .glb file. I don’t completely miss the point of being able add buffers as separate binary file, but that just increases the number of files to be managed. Which kind of represents the problem with GLTF in general is that it tries to please everyone. When really if you needed to manage a scene you could use a .blend file. If you need a lot of rigging and complex camera movement you could use an .mmd file. There’s really no need to reinvent the wheel for either of these. The weakest link when it comes to 3d files is that you either have files like .OBJ or .PLY that can’t use bones, or you have to go up to a .DAE file which requires you define a freaking scene. What’s needed is a simple asset format. The Khonos group even specifies this in their introduction to GLTF:

The GL Transmission Format (glTF) is an API-neutral runtime asset delivery format. glTF bridges the gap between 3D content creation tools and modern graphics applications by providing an efficient, extensible, interoperable format for the transmission and loading of 3D content.

Asset Delivery Format, I couldn’t tell with all these freaking nodes. But back to the main problem with accessors and buffer views, is they basically do the same thing. You have a buffer that contains all of the data required to define all of the vertices and faces in the file. And then from the mesh you point to an index for an accessor that gives the range, format, min and max, and which buffer view to use and then the buffer view says which buffer, from where to where and how long the buffer is. You’re basically adding two needless layers for something that can just be done from the mesh. “Here’s the position, it’s in this buffer, start at this offset, use this stride, this component type and this min and max”. At best I could understand just having buffer views with this information unique to each buffer view, but accessors and then buffer views? No, it’s stupid and redundant.

Materials are Stupid

    "materials": [
        {
            "pbrMetallicRoughness": {
                "baseColorTexture": {
                    "index": 0
                },
                "metallicFactor": 0.0
            },
            "emissiveFactor": [
                0.0,
                0.0,
                0.0
            ],
            "name": "blinn3-fx"
        }
    ],
    "textures": [
        {
            "sampler": 0,
            "source": 0
        }
    ],
    "images": [
        {
            "uri": "DuckCM.png"
        }
],

Not sure why all materials need to have "pbrMetallicRoughness" definition, but you do and it's stupid. Also I'm not sure why "baseColorTexture" needs to be declared inside the metallic definition. Materials would make a lot more sense if this were not needed, or if you could just declare phong shading, or your own custom shaders somewhere. I'm not sure why everything needs to have a metal definition, or that's the trend the industry is taking and everything needs to be shiny and reflective. Could be and I could just be an idiot, but I would like to be able to declare some simple models without all of this non-sense. Images and textures seems redundant, but I guess not that bad. You can declare images as a list of images, and then you can declare textures with different sample, clamp or mirror settings.

Asset Definition

{
	type : "mesh",
	position : {
		buffer : 0,
		offset : 0x1200,
		stride : 0x12,
		count : 100
	},
	normal : {
		buffer : 0,
		offset : 0x1200,
		stride : 0x12,
		count : 100
	},
	bones : {
		buffer : 0,
		offset : 0x100,
		count : 5
	},
	weights : {
		buffer : 0,
		offset : 0x2000,
		count : 100
	},
	calls : [
		{
			material : 0,
			indices : {
				buffer : 0,
				offset : 0x3000,
				count : 30
			}
		},
		{
			material : 1,
			indices : {
				buffer : 0,
				offset : 0x3400,
				count : 23
			}
		}
	],
	animations : [
		{
			buffer : 0,
			offset : 0x4000,
			name : "run"
		}
	]
}

So what's the practical application of all my ranting? Have all of the definitions inside the "mesh" node. If you define the type as a mesh, then you can make a lot of definitions implicit. Do I need to tell the accessor that I'm looking for a set of three floats for each position. No because it's the freaking position, that should be implicit. In what situation are you going to have any other data between a single set of x, y, z coords? The answer should be never. If you make definitions for bones and animation, then you can throw those in the buffer as well, since that not something that you really need to edit. Like if you have a mesh as a child of another mesh, then I could understand why you might want to adjust the position by hand. But for complication meshes with hundreds of bones, is that something that should be taking up space as text in the file? Definitely not.

For calls you can either do things one of two ways. You can either have a buffer where you have everything ready to be copied to the graphics memory for performance, or the style above that keeps positions separate to be loaded into a texture editor. And you can assume that the indices are going to be unsigned shorts in pairs of 3 for triangle faces and again, don't need to define the type, because it should be implicit. So rather than having a ton of different data types and indexes cross referencing and redefining the same redundant information, you can keep everything in one short view to be easily read and exported by different applications, rather than defining a file type that no one can support.

Getting Reorganized

In order to consolidate my Raspberry Pi’s I bought the Gekkworm x830 case, and already I’ve having second thoughts on the purchase. The case itself does what it’s advertised to do, which is basically house a Pi with a 2.5″ hard drive. The downside is I bought a 5400RPM 2TB hard drive for it and I’ve forgotten about how stupid physical medium is, so I might have to open it up and replace the hard drive with my left over 256GB 2.5″ SSD drive.

After that the plan was to set up the Pi as the access point for all of my devices. So I’ll need to set up the wifi so that my other devices use it to connect to. Also one thing I’m not sure about with the aluminum case is the resulting on board wifi, so it’s probably a good idea to use a usb adapter to get better signal strength. One thing I’m not sure about is the wifi drivers for most of the usb devices, so I’ll have to look into which devices have which drivers, and then order a new usb dongle for that.

After setting up the ssd and the wifi dongle. The next step will be to set up nginx to host any projects I might have on it. Specifically that will mean editing the dns settings so that I can add dns settings for the devices I connect to it. And specifically for the case of nextcloud, have a cloud server to manage validating a certificate with let’s encrypt and then syncing that with my home router.