LuaJIT ffi, reference to union member array

[grump]

Anyone knows whether or not this is allowed in LuaJIT?

Code: Select all

local union = ffi.new[[
	union {
		uint8_t u8[8];
		...
		uint64_t u64;
		double n;
	}
]]

local u8 = union.u8 -- is this allowed? is it a valid and accessible array reference?
...
u8[0], u8[1] = ... -- access via the local reference, gives strange behavior
union.u8[0], union.u8[1] = ... -- works as expected, no strange behavior

I'm getting unexpected results with code that does this. No errors, but strange and undeterministic behavior, like the values not always updating when writing to u8[x].
If I use union.u8 instead of the local reference, everything's fine.

[Xgoff]

What you're doing with this union (writing to one field and reading from another) is called type punning, and you have to be really careful when doing it; LuaJIT has strict rules around doing things like this (to allow better optimizations), and the behavior you're experiencing is a result of that. Basically, the FFI's rules don't always line up with those of Lua itself; this is one of them.

LuaJIT allows type punning, but only if it's explicitly done via a union access; using raw pointers (or arrays or whatever) for this purpose isn't allowed. By doing just `u8[0]`, for example, the JIT only sees an access to the raw pointer and "forgets" that it actually came from a union. Oops. By doing `union.u8[0]`, the JIT can now see the whole union access and therefore allows the type punning, which then works as expected.

In general, you don't want to localize cdata lookups like this, anyway; the JIT is perfectly capable of doing that on its own. In fact it was designed to do it on its own!

[grump]

What you're doing with this union (writing to one field and reading from another) is called type punning,

I'm not doing type punning in the example though, just plain writing, and (later) reading the same field.

LuaJIT allows type punning, but only if it's explicitly done via a union access; using raw pointers (or arrays or whatever) for this purpose isn't allowed.

I read the documentation several times in search for an explanation for the behavior, because I feared I might rely on undefined behavior, but did not find anything that said "don't do this". Or maybe I have trouble understanding it. Can you show me where this rule is documented?

Are you refering to this:

The JIT compiler implements strict aliasing rules: accesses to different types do not alias, except for differences in signedness (this applies even to char pointers, unlike C99). Type punning through unions is explicitly detected and allowed.

?

In general, you don't want to localize cdata lookups like this, anyway; the JIT is perfectly capable of doing that on its own. In fact it was designed to do it on its own!

I'm not doing it because the JIT isn't capable. I wanted to do it for code brevity reasons. It's just tedious having the fully qualified name appear 8 times in a row.

[Xgoff]

I'm not doing type punning in the example though, just plain writing, and (later) reading the same field.

If you're talking about your first post... well there's no reading going on. Anyway, I assume this thread has something to do with your Blob library, and you certainly are doing quite a bit of type punning in that. I guess you tried to eliminate the `_native.` prefixes and started running into this problem?

I read the documentation several times in search for an explanation for the behavior, because I feared I might rely on undefined behavior, but did not find anything that said "don't do this". Or maybe I have trouble understanding it. Can you show me where this rule is documented?

Are you refering to this:

The JIT compiler implements strict aliasing rules: accesses to different types do not alias, except for differences in signedness (this applies even to char pointers, unlike C99). Type punning through unions is explicitly detected and allowed.

?

Yes. Unfortunately the "through unions" part is kind of vague; a Lua programmer might think there's no difference between `t.x[0] = ...` and `local x = t.x; x[0] = ...` for cdata objects, because with typical Lua objects there really is no difference. But the optimizer treats the former as a special case for unions, which can lead to a difference.

In fact, even the typical `x, y = y, x` idiom doesn't necessarily work for cdata as you'd expect in some cases.

I'm not doing it because the JIT isn't capable. I wanted to do it for code brevity reasons. It's just tedious having the fully qualified name appear 8 times in a row.

I mean, you can localize cdata lookups (except when punning), but this actually makes things more difficult for the JIT... except in rare cases. So you usually don't, but it's your call.

[grump]

If you're talking about your first post... well there's no reading going on. Anyway, I assume this thread has something to do with your Blob library, and you certainly are doing quite a bit of type punning in that. I guess you tried to eliminate the `_native.` prefixes and started running into this problem?

Yes, but not only the _native access was showing weird behavior. I changed this

Code: Select all

function Blob:readU64()
	local ptr = self._readPtr
	self._readPtr = ptr + 8
	return self._endian_in._64(self._data[ptr], self._data[ptr + 1], self._data[ptr + 2], self._data[ptr + 3],
		self._data[ptr + 4], self._data[ptr + 5], self._data[ptr + 6], self._data[ptr + 7])
end

to this:

Code: Select all

function Blob:readU64()
	local data, ptr = self._data, self._readPtr
	self._readPtr = ptr + 8
	return self._endian_in._64(data[ptr], data[ptr + 1], data[ptr + 2], data[ptr + 3],
		data[ptr + 4], data[ptr + 5], data[ptr + 6], data[ptr + 7])
end

After this simple change tests started to fail occasionally, with symptoms and error locations randomly changing in each run. Once I removed access through local aliases, everything worked fine. self._data is a ctype (uint8_t*) btw.

Anyway, thanks for your explanations. I will keep this limitation in mind for future projects.