special character in filename cause the filename to appear blank in SMB2 discovery

I am trying to access files from a smb share. Inside my test folder, I have two files named test.mp4 and test.mp4.

If accessed from iOS, the test.mp4 name will be blank as shown in the following picture. And I cannot interact with that file.

But the same file will appear normal on a android device.

I am using version 3.2.13(375) for iOS and version 3.3.4 for android.

assigned to @fkuehne

moved from #25945 (moved)

moved from vlc-ios#1165 (moved)

assigned to @fkuehne

added libvlc/Core in VLC-iOS label

Still reproducible with VLCKitv3 HEAD.

So this issues is caused by the UTF-8 conversion in libsmb2 which writes high/low surrogates for UTF-8, which is invalid. It was fixed when the implementation was rewritten in https://github.com/sahlberg/libsmb2/commit/bd2d166a7f54cfaa6d6e4a4c70f4430cbc673dee#diff-ad18986d219ad0ea1454f603a549e559768d1b180e9dc1dd8c215339416e25d2

That said, VLC should validate the filename is actually valid UTF-8 and reject it or replace the wrong characters if it's not, as libvlc users probably expect valid UTF-8 strings.

Moving back to the mainline vlc bug tracker as the problem is not specific to iOS - it just does not show android due to an implementation detail of the OS.

changed title from special character in filename cause the filename to appear blank in iOS app to special character in filename cause the filename to appear blank in SMB2 discovery

added Component::Services discovery label

assigned to @ePirat and unassigned @fkuehne

In case anyone is curious, here is the full investigation on this issue:

The data

The file in question has the name test🏮.mp4, the correct UTF-8 representation for this would be:

{ 0x74, 0x65, 0x73, 0x74, 0xF0, 0x9F, 0x8F, 0xAE, 0x2E, 0x6D, 0x70, 0x34, 0x00 }

but the actual string we get back from libsmb2 is:

{ 0x74, 0x65, 0x73, 0x74, 0xED, 0xA0, 0xBC, 0xED, 0xBF, 0xAE, 0x2E, 0x6D, 0x70, 0x34, 0x00 }

So let's have a closer look at whats in there:

• 0x74, 0x65, 0x73, 0x74 are test, as expected
• 0xED, 0xA0, 0xBC now this is the more surprising part, 0xED hints this might be a surrogate. Specifically this is the UTF-8 representation of the U+D83C high surrogate.
• 0xED, 0xBF, 0xAE this is the UTF-8 representation of the U+DFEE low surrogate.
• 0x2E, 0x6D, 0x70, 0x34 is .mp4, as expected again

Surrogates

Surrogates are a way to address characters beyond the BMP (Basic Multilingual Plane) used in UTF-16, a high surrogate followed by are low surrogate. The two surrogates are joined to get the actual unicode character. More information about this can be found in the Unicode Book.

Now if we would combine our two surrogates, U+D83C and U+DFEE, we get U+1F3EE which is our familiar lantern .

However, surrogates are not necessary and actually invalid in UTF-8 according to Table 3-7 in the The Unicode Standard:

Table 3-7. Well-Formed UTF-8 Byte Sequences

Code Points	First Byte	Second Byte	Third Byte	Fourth Byte
U+0000..U+007F	00..7F
U+0080..U+07FF	C2..DF	80..BF
U+0800..U+0FFF	E0	A0..BF	80..BF
U+1000..U+CFFF	E1..EC	80..BF	80..BF
U+D000..U+D7FF	ED	80..9F	80..BF
U+E000..U+FFFF	EE..EF	80..BF	80..BF
U+10000..U+3FFFF	F0	90..BF	80..BF	80..BF
U+40000..U+FFFFF	F1..F3	80..BF	80..BF	80..BF
U+100000..U+10FFFF	F4	80..8F	80..BF	80..BF

So when trying to construct a CFString (macOS/iOS CoreFoundation's string type) it fails and the item gets no name.

The actual issue

The issue was with libsmb2's ucs2_to_utf8 function. This was fixed when it was rewritten recently in bd2d166a7f54cfaa6d6e4a4c70f4430cbc673dee.

The library seems to be using CESU-8 where VLC expects UTF-8. It wouldn't necessarily be a bug if the library names and documents stated it. But it seems that they claim to use UTF-8? If so it is evidently a library bug.

...

-- Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.

Yeah it's a bug according to https://github.com/sahlberg/libsmb2/issues/207

You could have spared writing and formatting the long-winded explanations... uBut ,well, what's done's done.

Yeah, unfortunately I only found the upstream ticket after that, would have saved me a bit of time investigating this… While I can bump the contrib so that the smb2 version has the fix, I am not sure what we should do about linux where typically no contribs are used?

what we should do about linux where typically no contribs are used?

You do nothing. configure version checks should be about the API, not the bug fixes, as distributions might back-port fixes.

And Debian does not even ship libsmb2.

Ok, thanks for the clarification.

added Component::Build system: Contribs label and removed Component::Services discovery label

mentioned in merge request !970 (merged)

closed with commit f8212993

closed with merge request !970 (merged)

mentioned in merge request !998 (merged)

mentioned in commit 1819e17f

See !998 (merged)

changed milestone to %3.0.17