SHA1 checksum fails on 12.X.X even though messages are equivalent

[koraykoska]

After days of digging into it and trying to understand what's going on, I have concluded that the 12.X.X checksums sometimes fail. It's deterministic but I don't know based on what.

I am talking about this line.

It happens for both text and binary cases. The Docker testsuite has been used (latest tag as of 4 August 2023). We tried both echoing frames or echoing whole messages. In both cases, some test cases fail.

The client definitely receives the correct messages and echoes them correctly back. All of that has been checked thoroughly. What's interesting is that most of the 12.X.X test cases work perfectly fine in all environments, some fail on some operating systems and others on others. But within the operating system, it's consistent and deterministic.

But the most interesting bit is that exactly equivalent frames (and messages) work in other test cases (as similar ones are repeated throughout the suite). Again, deterministic.

Without knowing too much about implementation specifics, I think one of 2 things are happening:

• This leads to different hashes and doesn't accept correct messages from the client as it'sthe other way round on the other end.
• It's a compression thing where the uncompressed message is correct but the testsuite checks the compressed message. There might be cases where deflate is not deterministic but the uncompressed data is still correct.

I don't know too much about the testsuite and the implementation details, so it's difficult for me to rule out either of those 2 ideas and also to come up with others.

Would there be any way to change the way this comparison works to fix this? Full message comparison (uncompressed) for example? Even if temporary, we would like to continue testing 12.X.X testcases.

If there is anything I can provide to help debug this, I will happily do this.

[oberstet]

For binary messages, the equivalence of messages is the identity, and hence hashes MUST match. For text messages, the echo'ing peer might do some wrangling / normalization allowed by Unicode, but this is still dubious and the actual messages would need to be analyzed. IOW: it MUST work with at least binary messages (hashes must matcH) otherwise your testee is bogus.

[koraykoska]

@oberstet For the binary messages I can 100% confirm that the resulting data is correct if I send the same data to a different server and make exact binary comparisons. Maybe there is an issue with the compression config on the client? Would the comparison fail if the compressed data is incorrect or is only the uncompressed data compared?

[oberstet]

the hashes are computed over the uncompressed message payload. I don't think this is the reason. My bet is your testee language (which one btw?) does some Unicode normalization/wrangling ... which might be allowed (Unicode has some "equivalence" definitions), but it might also be invalid/disallowed, and then your testee is buggy. Anyways, more investigation would be needed to find out ..

[koraykoska]

@oberstet It's a thin Swift wrapper on top of libwebsockets. I did not try to run libwebsockets directly through Autobahn as it's pretty battle tested already. The Swift implementation is just a wrapper and doesn't do any data manipulation.

I don't know where to start to be honest with debugging as simply replicating the test suite with a Javascript websocket server leads to correct results. Would the reports help? I can upload them.

[koraykoska]

Just to repeat the interesting bit: It's dependent on the OS but deterministic. On macOS 13.0+ all of the tests always pass. On Linux (Ubuntu 20.04 and 22.04) with Swift 5.7 some fail, with Swift 5.8 others fail.

[koraykoska]

@oberstet

This is Swift 5.8 on Ubuntu 22.04. Case 12.3.18 is the only 12.X case that's failing here. Might help to debug.

https://transfer.sh/c8BbWePeow/autobahn-libwebsockets-swift.tar

[oberstet]

Would the reports help? I can upload them.

thanks, but I don't have time to dig in ..

On macOS 13.0+ all of the tests always pass. On Linux (Ubuntu 20.04 and 22.04) with Swift 5.7 some fail, with Swift 5.8 others fail.

this sounds like a bug in swift

I don't know where to start to be honest with debugging

write a simple swift program to do the following:

1. serialize unicode strings (you need fancy strings) on system A into bytes
2. copy the bytes to system B
3. deserialize bytes into string
4. and then reverse back to system A

data must roundtrip to the single bit.

stuff like this. what I want to say: under the hypothesis that this is some swift implementation bug related to fancy unicode strings and normalization/whatever-wrangling, you should be able to prove that using a simple swift program without any websocket or what.

hope this helps

[oberstet]

should note: I'd do the simple roundtrip (unicode=>serialize=>deserialize) both with two different systems A and B (those you mentioned above), and do roundtrip on A-A and B-B (self roundtrip) ... actually, the roundtrip could work on A-A but not on B-B ... for certain unicode strings only that is (it might work with simple us-ascii all the time etc)

so follow a systematic bisection approach to identify the issue in a simplified setting (no websocket or what, just plain swift programs and text/binary)

[oberstet]

This is Swift 5.8 on Ubuntu 22.04. Case 12.3.18 is the only 12.X case that's failing here.

mmh. the test data used in those tests is here

https://github.com/crossbario/autobahn-testsuite/tree/master/autobahntestsuite/autobahntestsuite/testdata

this is used to generate the 12.* test cases

autobahn-testsuite/autobahntestsuite/autobahntestsuite/case/case12_x_x.py

Line 57 in 09cfbf7

WS_COMPRESSION_TESTDATA = {

and test subscases 12.N.*

https://github.com/crossbario/autobahn-testsuite/blob/09cfbf74b0c8e335c6fc7df88e5c88349ca66879/autobahntestsuite/autobahntestsuite/case/case12_x_x.py#L231C13-L231C22