RediSearch AIX portability - I had a go, but (hopefully) small thing in the way

I am interested in the AIX portability of RediSearch, and am trying to get it working on AIX 7.1 x64.

I got Redis 4.0.6 up and running pretty smoothly.

I ran into a few issues with RediSearch 1.0.1 (1.0.2 was released a few minutes later :wink: ), see here for my efforts so far: .

So, it does something, and even indexes the documents, but FT.SEARCH finds nothing.

Any idea’s on what’s wrong? Can’t put my finger on it. Any help appreciated.

Cheers, TW

This looks like a problem with the encoder/decoder of the inverted index, it uses various schemes of integer encodings, we’re probably relying on endinanness somewhere and this causes the decoded records to be junk.

Good info, could well be endianness (AIX=BE).

Perhaps I could do an extra test on Linux zseries (like RHEL on IBM Power), which is Linux big endian. Will take some time to set up though, and has to be worth the effort.

It could also be the uint128 bitmask for field ids.
Mark made a chagne to that in the branch you just tested today, you might want to try that on AIX.

Thanks, doesn’t hurt to try. I just merged dmd-improvements into the master of .

Alas, same behaviour.


FT.ADD “ftFoo” “ftFoo\x00Bar” 1.0 ‘REPLACE’ ‘FIELDS’ x ‘hello world’

(used null character as extra)> FT.SEARCH “ftFoo” world

  1. (integer) 0> keys *

  1. “idx:ftFoo”

  2. “ftFoo\x00Bar”

  3. “ft:ftFoo/world”

  4. “ft:ftFoo/hello”> FT.INFO ftFoo

  1. index_name

  2. ftFoo

  3. index_options

  4. (empty list or set)

  5. fields

      1. x
  6. type

  7. TEXT


  9. “1”

  10. num_docs

  11. “1”

  12. max_doc_id

  13. “1”

  14. num_terms

  15. “2”

  16. num_records

  17. “1.8446744073709552e+19”

  18. inverted_sz_mb

  19. “17592186044416”

  20. offset_vectors_sz_mb

  21. “1.9073486328125e-06”

  22. doc_table_size_mb

  23. “5.626678466796875e-05”

  24. key_table_size_mb

  25. “4.1961669921875e-05”

  26. records_per_doc_avg

  27. “1.8446744073709552e+19”

  28. bytes_per_record_avg

  29. “1”

  30. offsets_per_term_avg

  31. “1.0842021724855044e-19”

  32. offset_bits_per_record_avg

  33. “8”

  34. gc_stats

    1. current_hz
  35. “1”

  36. bytes_collected

  37. “205”

  38. effectiv_cycles_rate

  39. “0.0058139534883720929”

Our “qint” AKA GroupVarInt codec assumes little endian. We can modify it to be big endian if we had some easy way to test it. consider

It copies the first byte of the uint32 while the integer’s value is not 0. This works in little endian because the first byte is going to be the the MSB; however in big endian the first byte is actually the LSB, so for example encoding the number 0xFAFBFCFD the following happens in little endian:

  1. Because of byte order, the number is actually 0xFDFCFBFA
  2. The first byte (0xFD) is written
  3. The number is right shifted, the byte layout is now 0xFCFBFA00
  4. The first byte (0xFC) is written
  5. repeat steps 3-4 until the number is 0

With big endian:

  1. The number in memory is 0xFAFBFCFD
  2. The first byte (0xFA) is written
  3. The number is right shifted, the byte layout is now 0x00FAFBFC
  4. The first byte (0x00) is written
  5. … as you can see, this doesn’t encode the correct number.

It should be simple enough to modify this algorithm to work on big endian machines - but as an ifdef. this algorithm is specifically optimized for little endian (because we can just read byte by byte sequentially)

Mark Nunberg | Senior Software Engineer
Redis Labs - home of Redis


I got LSB and MSB confused, but you get the idea :slight_smile:
Mark Nunberg | Senior Software Engineer
Redis Labs - home of Redis


Thank you Mark, I kind of suspected a construct like that after Dvir’s remark.

If I’m not misunderstanding, I could try to do about the same in BigEndian, but decrement instead of increment the pointer.

For the ifdef in this scenario, would you use something like this?

gcc has predefined compile time macro variables which take care of determining the byte order:

As far as the best strategy for fixing this. is where the number is actually decoded. As you can see, it assumes that the encoded integer is actually a uint32_t (in memory). This part would also need to be changed, because reading the value “0x01” would mean reading all four bytes, which would cause garbled data at best and invalid memory access at worst. (FWIW I’m not 100% comfortable with this kind of type casting anyway, because technically it might cause a buffer overrun anyway).

What would work best here would be to “normalize" the number into little endian before handling encoding/decoding

In summary, the following changes would need to be made:

  1. In __qint_encode, if big-endian, write the last rather than first byte of the uint32 (converting it to little endian in-situ)
  2. in QINT_DECODE_VALUE, rather than using magic casting, you’d need to combine bits manually. Because the encoding is now always LE, you don’t need to ifdef here, but simply combine bits with this understanding. Hopefully the compiler will still optimize for LE

switch (bits) {
case 2:
lval = *(uint32_t *)ptr & 0xFFFFFF;
nused = 3;
break; \

would become:

case 2: lval = (ptr[0]) | (ptr[1] << 8) …

case 3: lval = (ptr[0]) | (ptr[1] << 8) | (ptr[2] << 16)

Or more generically

nused = 4-(bits-1)

for (size_t ii = 0; ii < nused; ++ii) {

lval |= (ptr[ii] << (8*ii))

or something like that. We have the loop “unrolled" for performance but it’s functionally equivalent to smaller code.

Mark Nunberg | Senior Software Engineer
Redis Labs - home of Redis


Excellent info, I’ll get back to you on this.