Both the boys have fallen asleep, so I am watching Akira and programming.
This time, I am rejigging the way I handle colour and UVs. Every polygons has a primitive attributes pointer that holds references to the original vertex data along with some colour, materials, etc.
When spitting out the final polygonal mesh, I can take all the colour and generate a colour map for the base diffuse lookup, or return them as vertex colours. I am also setting up the UV sets for each reference vertex as an array basically so I can apply as many UV sets as wanted or any feature might require.
But also having a fixed colour palette for the base diffuse that gets passed into the mesh generation to convert the colour values to UV values for the predefined palette.