YouTube "n" parameter scrambling prevents playback (former throttling issue)

I'm experiencing the same issue. Appears to be the same throttling that has popped up in the past in this thread:

Youtube.lua likely needs an update.

I confirm this occuring to me as well. VLC version: 3.0.8 Vetinari

Attached are the debug messages for only the first 18 seconds of this video: https://www.youtube.com/watch?v=mBf6kJIbXLg ... couldn't wait more, video was almost unwatchable, even when lower resolutions were set through preferences ...

a lot of warning messages 'Couldn't parse unidentified YouTube video throttling parameter transformation' in the beginning, and after starting displaying some frames
a lot of 'Buffering xx%'
'picture is too late to be displayed (missing xx ms)' and of course
'buffer too late (xxxxxx us): dropped' messages

The throttling warnings are coming from the lua descrabling fuction.

Pls note, that if the same video is played via the url coming from yt-dlp executed with the --get-url argument, there are absolutely no hanging/freezing/dropping issues.

Hope it gets fixed.

Dbg_vid_Id_mBf6kJIbXLg.txt

@linkfanel Can you help us here?

@lance78 Have you had any success fixing this issue? I'm still experience throttling for YouTube urls as of 8/19/22

Only reported the issue is this ticket opened for VLC to investigate and potentially provide a solution. Throttling is still occurring. I believe that this ticket has to be assigned.

@Courmisch Can you assist with assigning the ticket? I recall you assigned a similar ticket to this one a few months ago (#26174 (closed))

I'm not sure what you're expecting of me here. The referenced bug is assigned to the reporter, which wass also @lance78 .

I rarely assign bug to people other than myself or the respective OP, as I'm not in a position to order people around.

I am not sure why ticket #27227 would have been assigned to me. I simply reported the issue. I am showing "0 assignees" in the status for this ticket. I humbly ask that the ticket be assigned to the one who normally works on the bug fixes for the lua script.

@chouquette @ePirat @jbk I wanted to inquire if any of you could point us in the right direction for who to assign this ticket to? Apologies for the call out but I saw you listed as contributors on the youtube.lua GitHub page.

assigned to @linkfanel

Sorry for replying late. It seems that something's wrong as I haven't received email notifications from GitLab since May, so I had wrongly figured there had been no ticket opened about this yet.

Anyway, it doesn't matter all that much as I only have bad news: this isn't going to get fixed. The new descrambling code features table indices represented as arbitrary arithmetic expression potentially including Math.pow() calls, and number represented as Date objects parsed from a wide range of various unspecified date string formats e.g. new Date("Wednesday 31 December 1969 19:00:53 CDT or new Date("1969-12-31T17:00:58.000-07:00") - and date parsing is explicitly implementation-specific in JavaScript. The descrambling instruction chain, instead of being simple and linear, is now seemingly complex with subcalls and branching using ternary operators and try-catch-finally statements among others. This would require an arithmetic engine, a compatible date parsing library, and especially an execution tree parser. I'm not writing custom support for that, because it might as well be a generic lightweight JavaScript interpreter: see #26318 As for transformation routines, there are new ones and I haven't really analyzed how much work it would be to support them, but that would be the easy part.

Since like yesterday I'm also hitting a new bug where playback completely fails because unexpected input crashes the script, but I can and will fix this oversight of mine. So that we can still sort of play low-resolution videos and still-frame music.

If you want to help, you can go to #26318 and suggest lightweight JavaScript interpreters, preferably available as a lua library, that we could use to rewrite this. Or you could provide further analysis about the new descrambling code, indicating ways in which it might be simpler than it looks or suggesting how to emulate it without a lot of work, but I'm skeptical.

Alternatively, you could contact your local Member of the European Parliament (or your local representative wherever you are, but the EU might be the best bet) to petition for regulations and enforcement to ban this kind of practices, which have as apparent intent and effect to discriminate and control which user agent software is allowed or not, hampering competition and consumer rights, and also violating net neutrality. You could also sue YouTube for violations of your interoperability rights or whatever applicable, but be aware that YouTube might then close your account or access for violating their (from this point of view unfair and invalid) Terms of Service.

The three URLs in the OP work fine with YT-DLP. Does this mean YT-DLP has a Javascript interpreter?

Yes: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/jsinterp.py

It apparently required updating to support the new code too: https://github.com/yt-dlp/yt-dlp/commit/8f53dc44a0cc1c2d98c35740b9293462c080f5d0

Well, yt-dlp also uses the "new" current Innertube YouTube API, and I hear that with the right client parameters you can request video URLs that don't need parameter descrambling, so I suppose we could get some mileage out of that. However that API is accessible only through HTTP POST, and we still don't have an HTTP POST API, and no other way to make HTTP POST requests from lua playlist script: so that would be #26185

We already support HTTP PUT and HTTP request bodies. The only missing bit for HTTP POST is formatting the payload data, which is not exactly hard.

I have a WIP branch adding HTTP APIs to Lua that might be useful for this, though currently no time to polish this enough for a MR yet.

The payload data here for Innertube is JSON. I'm not sure how much of an immediate need we have for supporting POST APIs that still consume form-encoded data rather than, say, something modern like JSON. And from a VLC API design point of view, I'm not sure an HTTP POST API needs a form data encoder, or knowledge of what the payload is beyond a blob labelled with a Content-Type. We do have a need for allowing the API to pass custom HTTP headers to the POST request.

Anyway, to help with this I'm still missing an HTTP POST lua binding in VLC 3.0.x, and I'm not going to contribute it, sorry.

Reminder that with HTTP POST lua bindings, we should be able to bring back twitch.lua too: #25397

If someone provides a JavaScript interpreter available as a lua module, I'm gladly offering to do the work to use it, that would be even simpler than HTTP POST. I'm not going to integrate into the build system or APIs a third-party interpreter in C or whatever, although that's another valid option.

VLC 3.0 has no support for HTTP payload requests anyhow, so you couldn't make Lua bindings if you wanted.

Personally, I'm happy with my external 3.0 backport of the 4.0 YTDL plugin, so I don't care one bit about the Lua implementation. Truthfully, it would help if youtube.lua was purged so I wouldn't have to delete it manually every time.

Please don't purge it.

I'm exploring a solution for youtube.lua. Will post if I make progress

VLC 3.0 has no support for HTTP payload requests anyhow, so you couldn't make Lua bindings if you wanted.

Okay, good point, so that orients options even more.

Personally, I'm happy with my external 3.0 backport of the 4.0 YTDL plugin, so I don't care one bit about the Lua implementation. Truthfully, it would help if youtube.lua was purged so I wouldn't have to delete it manually every time.

Well that's not terribly helpful, I'm happy too to make a lot of hacks on my own system so I personally won't be affected by issues average users will still have to struggle with. What's the problem with properly adjusting module priorities to take precedence over youtube.lua without having to delete it though?

Now if I understand correctly, you're not really suggesting a YTDL plugin backport as a solution for everyone, either in tree or out of tree. But for the sake of clarity, let's discuss that possibility. Obvious issues with that would be that nobody wants to deal with an out-of-tree module; it's really troublesome in tree to backport into a stable release something that's going to hijack current functionality depending on whether or not some third-party software is installed; does it even backport easily with all the demux/access filter core changes there were?

And maybe the biggest issue, YTDL requires Python and is not shipped on all platforms, namely the Makefile.am excludes it from Windows, Android and... Emscripten? What platform does that correspond to exactly? I think that yt-dlp offers Windows builds, but I'm unclear whether ytdl-extract.py can tap into that Python install, or into the yt-dlp module installed from those builds. I would say that no Windows or Android support for this solution is a show-stopper.

Maybe I misunderstood and you want to put forward clearer plans for a 3.0 YT-DL plugin backport. But I think we're down to a JavaScript interpreter. If not one in lua, for 3.0 we always have the option to slap a copy of the interpreter's sources into modules/lua/ and directly link the lua binding calls to it. We could also try to transpile an interpreter into lua (or C), for example yt-dlp's jsinterp.py. Would the GPL be satisfied with shipping transpiled sources?

I'm exploring a solution for youtube.lua. Will post if I make progress

What's the solution you're exploring? All ideas welcome.

We could also try to transpile an interpreter into lua (or C), for example yt-dlp's jsinterp.py.

I found https://github.com/dmitrii-eremin/python-lua (off https://github.com/hengestone/lua-languages). Anybody with a setup to run random code from the internet wants to try if it works on https://raw.githubusercontent.com/yt-dlp/yt-dlp/master/yt_dlp/jsinterp.py ?

I have no plans to propose the plugin to 3.0 as I can't do the massive QA that it would deserve.

As for 4.0, although we could easily get rid of the intermediate Python script, I will not be making the module work on Windows for the simple reason that that was already explicitly rejected in the initial code review.

I don't think we have Lua on iOS and on WASM, so I don't really get your point. All I'm saying is that:

I have a better alternative to youtube.lua for my purposes.
I would rather youtube.lua be removed (or disabled from build) if it's not going to be fixed.

I have no plans to propose the plugin to 3.0 as I can't do the massive QA that it would deserve.

Right. I recall there were issues with the design that required changes to the demux core. Probably best not to merge it, okay. So I guess the solution would be a JavaScript interpreter.

As for 4.0, although we could easily get rid of the intermediate Python script, I will not be making the module work on Windows for the simple reason that that was already explicitly rejected in the initial code review.

Was it really? I went back and read the initial code reviews. As I remembered, this was the only such comment I could find: https://mailman.videolan.org/pipermail/vlc-devel/2020-September/138542.html

But as long as it never lands on Windows I don't really care.

And that's a complete non-argument, marred in personal drama. As far as I know there has been no real and rational rejection about the YTDL plugin on Windows. But I understand if that's reason enough for you not to work on it.

I would rather youtube.lua be removed (or disabled from build) if it's not going to be fixed.

It still does offer useful functionality as it is. And anyway we are not removing one module just because it fails to play nice with another one: if there is a priority or probing issue, please file a bug so it can be fixed. I mean, as a maintainer I could try and proactively fix what I'm guessing your grievance is, but it would go a lot better if you just filed a bug about it :)

What's the problem with properly adjusting module priorities to take precedence over youtube.lua without having to delete it though?

It already has higher priority, and it does determine the correct URL. But then the Lua stream filter still comes afterwards, matches the URL, hijacks the stream and causes a complete failure. I don't see how I can fix that with priorities.

This is just one example why I think broken stuff should either be disabled (if there's something to salvage for later) or removed (if not).

Was it really [explicitly rejected]?

I think it's fair to say so. I'm not going to spend time to develop a contentious MR for a feature that I don't care about (namely YT-DLP on Windows); there are already enough things to spend time on and to argue about as things stand.

I understand if that's reason enough for you not to work on it.

Ack.

I gave a try to https://github.com/dmitrii-eremin/python-lua to convert yt-dlp's jsinterp.py to lua. First of all it apparently lacks support for some Python keywords used by jsinterp.py related to exception handling such as raise, try or except. Then it doesn't support the from <module> import syntax - which wouldn't be such a big deal; and while it does support import statements, it merely converts them to a lua require for an inexistant lua module.

Making this work would require rewriting parts of jsinterp.py to do away with exceptions (or implementing support in pythonlua), and also inlining a bunch of external module code into jsinterp.py. jsinterp.py uses only limited utility code from elsewhere in yt_dlp so that part is doable, and some Python standard library code: mostly ChainMap data structures, the zip_longest iterator, and then the JSON module and regexp engine, and... those parts are less straightforward to inline. And then there's still no guarantee that it would work in the end.

I still think yt-dlp's jsinterp.py would be a good target to track; so remaining options would be rewriting it in lua by hand, or transpiling/compiling it into C / object code and then linking the lua bindings against it - for lack of a more ambitious VLC API architecture. I'm sure there is better technology around for turning Python scripts into object code than into lua. Help very welcome: see #26318.

Running Python interpreter in the VLC process is a no go. I tried already, but CPython simply isn't designed to work in multi-threaded and modular processes.

And you can't generally speaking compile Python. It is an interpreted language by design, with introspection and all.

Porting code manually may be possible but if you're implying that it's too painstaking for you, I doubt anyone will do it.

Running Python interpreter in the VLC process is a no go. I tried already, but CPython simply isn't designed to work in multi-threaded and modular processes.

Good to know; I don't think anybody really wanted to embed the Python VM into VLC anyway.

And you can't generally speaking compile Python. It is an interpreted language by design, with introspection and all.

There seems to have been at least one attempt: https://gcc.gnu.org/wiki/PythonFrontEnd

Gccpy serves as a prototype and work in progress on how GCC can provide a solid platform for compiler development, not just for low-level languages but for anything. Gccpy differs in approach to other python implementations such as pyrex/cython/shed-skin. This is a branch new fresh approach to python and complete re-implementation of everything, we have no dependency on libpython.so

It doesn't look alive though.

Porting code manually may be possible but if you're implying that it's too painstaking for you, I doubt anyone will do it.

Well I have practically NO experience with Python software development, and no experience with language compilers or virtual machines myself, so.

mentioned in merge request !2464 (merged)

mentioned in issue #27091

marked this issue as related to #27091

added Status::duplicate label

closed

removed Status::duplicate label

reopened

marked this issue as related to #27311 (closed)

Not the same issue as #27091 Well #27091's reporter hasn't confirmed the working hypothesis, but there are indeed two different issues here: data throttling, and fetching playback URLs from APIs at all for age-restricted videos.

mentioned in issue #27311 (closed)

Hi, I am a bit lost on this matter : do I understand correctly that the youtube.lua file is not going to be maintained ? I just downloaded VLC, and still have this throttling issue with last version of youtube.lua.

How do you recommand to stream youtube video now if VLC doesn't do it anymore ?

Thanks, Romain.

As noted above, I just delete youtube.luac. Then the YT-DL plugin deals with it just fine for me.

(Though TBH, I mostly use Youtube through the browser or my TV.)

Thanks for your reply. But this mean we can't stream the video anymore. Do you know any software that would still be able to stream rather than download a copy ?

Uh, works for me.??

Although at the moment we have not implemented a solution to the throttling issue, youtube.lua is still maintained, and provides partial functionality. Passing --preferred-resolution 0 or setting the corresponding setting in the preferences to the minimum, will tell VLC to always request the lowest bitrate from YouTube and help mitigate the throttling issue. youtube.lua still provides satisfactory functionality for music and other mostly-audio content, or if you're willing to be patient - YMMV. I hope this helps.

Thanks Pierre ! What is YMMV ?

I am also wondering : is it possible to find by hand the throttling parameter of a youtube video ?

If it is, we could arrange to let the user pass the throttling value as a parameter to VLC (?).

Another question : is there a VLC 4.0 on its way ? If so, would the youtube issue be fixed in this new version of VLC ?

YMMV = Your Mileage May Vary https://en.wiktionary.org/wiki/your_mileage_may_vary

I am also wondering : is it possible to find by hand the throttling parameter of a youtube video ? If it is, we could arrange to let the user pass the throttling value as a parameter to VLC (?).

We are not going to do that. But the way it works is that youtube.lua converts a typical YouTube page URL into a direct URL to an actual video stream that VLC can play. You can always pass such a direct stream URL, that you got externally, to VLC. For example if you use the network trace debugging console of your web browser on the YouTube website, to capture a non-throttled stream URL, and open it in VLC, it will work. You can also do this in many ways, for example getting the throttled URL returned by youtube.lua (from the verbose logs or the Ctrl+I panel), editing the URL to fix the throttling parameter, and then opening the fixed URL in VLC. With a little bit of setup, manually fixing the URL could be as simple as doing a bit of copy-pasting back and forth from a web browser's javascript console; but again that's something you'd have to do manually every time you play a video.

Another question : is there a VLC 4.0 on its way ? If so, would the youtube issue be fixed in this new version of VLC ?

My maintenance policy is that there is only one up-to-date youtube.lua, so the latest version is the same you'd get on 3.0, 4.0, or the development branch. Like Rémi said, there is a new, alternative plugin to play YouTube (and more) in 4.0, but it relies on the external youtube-dl tool (and Python runtime), it hasn't been widely tested by the masses, and it's not enabled on Windows for example. It would fix the throttling issue though.

added Component::LUA scripts label

mentioned in merge request !3272 (merged)

YouTube "n" parameter scrambling prevents playback (former throttling issue)

Child items 0

Activity