You're asking too much from floating-point precision. What you seem to be trying to achieve is typically done with a model matrix (or quaternion) that gets renormalized every while, to avoid this kind of drift. In your case the drift is always towards shrinking, but it can be either way.
The idea is: you have a model that is always in the same position. For example, a cube centred on origin and with side 24 would have vertices at coordinates {-12,-12,-12}, ...(all combos of signs)..., {12, 12, 12}. These numbers would be fixed, as they define the geometry of the model with respect to the origin, not its actual position or orientation.
Then you have either an orthogonal matrix or a quaternion (your choice) that represents the orientation of your model. That's the model matrix. After rotating the matrix or the quaternion, a small drift may happen, so every while you renormalise it again. For example, you check the squared norm of the quaternion, or the squared norm of each column of the 3x3 matrix (the first three rows and columns, if you use a 4x4 matrix that includes displacement), and if they differ from 1 in more than a certain threshold, you renormalise.
To represent the object, you apply the model matrix to every vertex. The vertex shader can do that for you, if you use a matrix (not sure if you use a quaternion) or you can do it in Lua.
Your application may be simpler and not need that much complication. If your object only rotates in that axis, and nothing can make it rotate in any other axis, you can just track the total angle, transforming the original model by that angle rather than the previous positions, using a formula similar to yours, except only the angle is treated incrementally:
Code: Select all
angle = (angle + math.rad(1)) % (math.pi * 2)
cos1 = math.cos(angle)
sin1 = math.sin(angle)
for i = 1, numPoints do
points[i].x = model[i].x * cos1 + model[i].z * sin1
points[i].z = model[i].z * cos1 - model[i].x * sin1
end