BD+ - Utilities for generating smaller cached convtabs via in-stream tables ($1754) · Snippets

Hi,

currently playing with these two utilities and it seems I'm a bit unlucky.

First, I managed to compile the tools by hooking them up into the build process of libbdplus (ie. add them to Makefile.am, then do the usual configure/make sdtuff).

The disc I'm trying this with is PROMETHEUS (MK 0498CB...) which according to bdplus_test has a BD+ SVM generation 14. Playback with the publicly available convtab and libbplus works fine.

So I created an ISO image of the disc using dd (dd if=/dev/sr0 of=bluray.iso) and mounted it somewhere, libaacs seems happy with this as it manages to decrypt using keys looked up from KEYDB.cfg. Next, I did

# bdplus_diff -e /iso/mount/point/ -c 0498C-convtab.bin -o 0498C-FUTout -d 0498C-diffout

which after processing the main title (00600.m2ts) ABORT()ed with "AACS failed" at 00500.m2ts (129024 bytes in size). I suspected some sort of copy protection measure (libaacs checks/verifies for a valid TS header mark which might just not be there, ie. invalid M2TS file/data), so I patched bdplus_diff.c with

@@ -158,7 +158,7 @@ static void *_init_aacs(const char *root, char *mk_str)
     return aacs;
 }
 
-size_t aread(FILE *fp, void *aacs, uint64_t offset, uint8_t *unit)
+size_t aread(FILE *fp, void *aacs, uint64_t offset, uint8_t *unit, uint8_t allow_fail)
 {
     fseek(fp, (offset - (offset % 6144)), SEEK_SET);
     size_t res = fread(unit, 6144, 1, fp);
@@ -172,7 +172,13 @@ size_t aread(FILE *fp, void *aacs, uint64_t offset, uint8_t *unit)
     }
     if (aacs && !aacs_decrypt_unit(aacs, unit))
     {
-        ABORT("AACS failed at offset %llx", offset);
+        if (!offset && allow_fail) {
+            printf("*** AACS failed at start of file on unit %u (copy protection measure?) ***", unit);
+            return 0;
+	}
+	else {
+            ABORT("AACS failed at offset %llx", offset);
+        }
     }
     return res;
 }
@@ -182,7 +188,7 @@ void read_obfut(FILE *fp, void *aacs, obfut_t *obfut_list, uint32_t *obfut_list_
     off_t curr_offset = 0;
     uint8_t unit[6144];
     uint16_t sp_id = 0;
-    while (aread(fp, aacs, curr_offset, unit))
+    while (aread(fp, aacs, curr_offset, unit, 1))
     {
         uint16_t skip = 0;
         for (int i = 0; i < sizeof(unit); i += 0xc0)
@@ -452,7 +458,7 @@ int process_convtab(char *disc_root, char *convtab_path, char *diff_output_path,
                                   (uint64_t)patch0_address_adjust) *
                                      (uint64_t)0xC0 +
                                  (uint64_t)patch0_buffer_offset);
-                aread(fp_m2ts, aacs, off0, unit);
+                aread(fp_m2ts, aacs, off0, unit, 0);
 
                 if (memcmp(&unit[off0 % 6144], patch0, 5))
                 {
@@ -465,7 +471,7 @@ int process_convtab(char *disc_root, char *convtab_path, char *diff_output_path,
                                   (uint64_t)patch1_address_adjust) *
                                      (uint64_t)0xC0 +
                                  (uint64_t)patch1_buffer_offset);
-                aread(fp_m2ts, aacs, off1, unit);
+                aread(fp_m2ts, aacs, off1, unit, 0);
 
                 if (memcmp(&unit[off1 % 6144], patch1, 5))
                 {
@@ -534,7 +540,7 @@ int compare_m2ts(FILE *fp_m2ts, FILE *fp_patched_m2ts, AACS *aacs, uint32_t file
     {
         if (unit_cnt % 1024 == 0)
             printf("Reading unit %llx of %llx - patch_count = %d\n", unit_cnt, total_units, patch_count);
-        aread(fp_m2ts, aacs, unit_cnt * 6144, unit);
+        aread(fp_m2ts, aacs, unit_cnt * 6144, unit, 0);
         fseek(fp_patched_m2ts, unit_cnt * 6144, SEEK_SET);
         fread(patched_unit, 6144, 1, fp_patched_m2ts);
         int16_t first_diff = 0;

That helped to let bdplus_diff complete it's job.

Then tried the second step with

./mask_regen -e /iso/mount/point/ -o 0498C-FUTout -d 0498C-diffout -m 0498C.segmask.bin

That determined quite a few keys and then suddenly stopped and locked up, last output:

...
sp_id212 - 156 non-overlapping/distinct exclusive patches - 326 relevant patches - 314 exclusive patches - patch1 key (80 occurences): f1d279547c
Title 00600 - sp_id212 - curr_final_key: 80ebd6f8870cfc4131cc25f1d279547c
Title 00600 - sp_id212 - good_patches = 160 - neutral_patches = 2, bad_patches = 0
sp_id213 - 71 non-overlapping/distinct exclusive patches - 158 relevant patches - 142 exclusive patches - patch0 key (1 occurences): 43f96dd0f4
getPatchKeys early ret - sp_id213 - type0 - occurences = 1
WARNING: Title 00600 - sp_id213 - type0 - getPatchKeys returned -1
sp_id213 - 71 non-overlapping/distinct exclusive patches - 158 relevant patches - 142 exclusive patches - patch1 key (73 occurences): fd6cd5b52b

mask_regen ran at 100% CPU (on one single core). I aborted after a few minutes with CTRL-C. Though the (partial) segmask let me play the disc up until to about the middle of the movie, the video then became garbled, and libbdplus DEBUG didn't mention itself providing patches from that point onwards, so obviously the segmask is incomplete.

Anything I may try?

Thanks! And thanks for your efforts!

EDIT: Note though that due to lack of spare time I didn't try another disc yet, this would be the next step for me to try, though I believe it should generally work with any disc.

Thanks for the feedback nst. I don't think you were unlucky - this code is currently temperamental. I've just updated the snippets here with a few fixes. I also experienced an infinite loop bug and it has been resolved now. Hopefully you were seeing the same issue. I recall having an AACS issue with one disc and it seemed to be due to my incorrect manual selection of the AACS unit key. Both utilities should now use the catch-all libaacs approach (title 0xFFFF), so hopefully this issue will also be resolved.

The only other change was that I noticed the convtabs for the American release of Firefly have several superfluous patches that don't seem to correspond to the BD+ in-stream FUT data. MakeMKV also has random data in these locations. The patches are around 10-15 consecutive bytes and they occur outside the actual video/audio data. The patch size alone violates the BD+ requirements as far as I can tell. I've adjusted the threshold for determining whether the current mask is acceptable. It is a hack, but it is only required for these 3 discs at the moment and resultant video still perfectly matches MakeMKV / the convtab (infinite PSNR measured with FFmpeg).

Just a quick heads up: Pulled in your changes and removed the AACS "hack", this causes bdplus_diff to bail out again (same disc as above):

processing sp_id389 - max = 391 - 519 obfuts
processing sp_id390 - max = 391 - 522 obfuts
processing sp_id391 - max = 391 - 1003 obfuts
Processing obfut for 00500.m2ts
 *** AACS failed at offset 0 ***
Aborted (core dumped)

Will put the hack back in and test further and report back later.

EDIT: Followup:

With the AACS allow_fail hack, bdplus_diff finished and produced identical output to what it did yesterday. mask_regen now finishes and quits normally, and it produced a segmask with 11738 bytes (segmask SHA256 hash: f0910e2aaad72bfb9da959615f59a71b5f48469892bb6de71cfdc47559570dd3). Playback testing needs to be done, I'm working remotely on the machine ATM ;) Though if you happen to own the same disc, maybe you can confirm if this segmask is correct by it's size and SHA256?

Again, thanks for all your efforts!

Thanks for testing it out. I'll need to look further into the underlying cause of that AACS issue.

Unfortunately I have a different copy of Prometheus (MK 2347...), so I can't confirm the validity of your segmask file. Would you mind uploading the full log from your testing of mask_regen to a site like pastebin?

One option for testing the segmask is blu-save by shironeko (https://git.sr.ht/~shironeko/blu-save). Because blu-save utilises libaacs/libbluray/libbdplus, it supports both full and in-stream (segmask) convtabs. I found it was necessary to add the line 'bd_play(bd);' at the beginning of the dump_dir function, otherwise libbluray didn't seem to properly initialise libbdplus (I'm sure a nicer fix would be possible). Once you have built the blu-save tool, run it once with the full convtab file in your bdplus convtab folder, and then replace the full convtab with the segmask from mask_regen. Hopefully the two decrypted/patched folders will be identical...

Sure, no problem - I reran the process for the Prometheus disc, the resulting segmask remains the same. Console output of mask_regen is at https://pastebin.com/zTThqsgp

On a side note, in the meantime I managed to generate segmasks for the remaining "newer-BD+-gen" discs I own (unfortunately thats only six discs 😄), and although I didn't test them yet, the process finished without any problems and the resulting segmask files appear sane (about 8-11kb in size), need to do playback testing later though.

Thanks for mentioning blu-save, been looking for some kind of copying tool involving libaacs+libbdplus. Unfortunately, without adding bd_play() it doesn't properly activate the convtabs nor the segmasks, and with bd_play() at the top of dump_dir(), the tool aborts with Java backtraces. I remember Kodi had issues regarding libbdplus in the past too (convtabs not being activated), though https://code.videolan.org/videolan/libbdplus/-/commit/d06af1580d69c09eb865f39cf0de7b544a776c99?merge_request_iid=3 helped getting this to work. Although that doesn't apply anymore since bdplus_m2ts() did change a bit, guess I have to fiddle around with that a bit to get blu-save to work.

Soooo...

finally managed to get blu-save to work by patching libbdplus (bdplus_m2ts() in src/libbdplus/bdplus.c) with this, which triggers BD+ startup and thus loads and activated a convtab or a segmask - whichever is available - if bdplus wasn't started by libbluray before (which is the case when not adding bd_play() in dump_dir()):

--- a/src/libbdplus/bdplus.c
+++ b/src/libbdplus/bdplus.c
@@ -325,6 +325,15 @@
 
     bd_mutex_lock(&plus->mutex);
 
+    if(!plus->cache_tab && !plus->conv_tab) {
+        /*
+            Try to get a cache_tab or conv_tab by firing BDPLUS_EVENT_START
+        */
+
+        BD_DEBUG(DBG_BDPLUS | DBG_CRIT, "[bdplus] bdplus_m2ts(%05u.m2ts): no conversion table, triggering BDPLUS_EVENT_START\n", m2ts);
+        bdplus_event(plus, BDPLUS_EVENT_START, 32, 0);
+    }
+
     if (plus->cache_tab) {
 
         st = segment_set_m2ts(plus->cache_tab, m2ts);

With that, I was able to get a decrypted and fixed (utilising libaacs and libbdplus) copy off the Prometheus disc with both the original convtab and the segmask generated using your tools.

Both copies were SHA256 identical! 😄

Will do the testing with the remaining discs aswell, very curious if all is well since mask_regen printed something about the key being 0000 on one or to segments on two of my discs.

I'm glad to hear that you got blu-save working and the original vs in-stream output was identical! And thanks for sharing your patch, it is appreciated. The message about a key being 0000... is generally benign (although I think libbdplus does produce a warning?). Certain segments may have no associated patches or the patches defined in the cached convtab produce no actual changes in the stream. In this case, the best/safest option is to output no mask. This can be the case during the opening (Eg: sp_id0) or closing credits of a title and some titles can have several such segments.

Well, the only "warning" I see with that regard is that one mask of any segmask file is missing, like

segment.c:1335: conversion table for 00600.m2ts does not have all masks (1/392 are missing)

but I get that on almost all discs being processed with a segmask file instead of a full convtab.

That said, I managed to verify all "newer gen" discs (ie. compare convtab vs. segmask handled BD+, see above) and for every disc, the output is identical. So, everything is alright and working fine, even with "Key = 0000" output 😄

Thanks!