Compare commits

...

52 Commits

Author SHA1 Message Date
fkrueger
db34fd2b52
Merge 4a70164539 into 4782a67038 2024-08-27 09:30:19 +02:00
Samantaz Fox
4782a67038
Release v2.20240825.2 2024-08-26 22:52:50 +02:00
Samantaz Fox
5baaedfa39
CI: Fix docker container tags (#4883)
Closes issue 4880
2024-08-26 22:48:14 +02:00
Samantaz Fox
4f066e880c
CI: Fix docker container tags 2024-08-26 21:55:43 +02:00
Samantaz Fox
3e17d04875
Release v2.20240825.1 2024-08-25 22:30:46 +02:00
syeopite
cec905e95e
Allow manual trigger of release-container build (#4877) 2024-08-25 19:55:52 +00:00
Samantaz Fox
80958aa0d8
Release v2.20240825 2024-08-25 21:25:48 +02:00
Samantaz Fox
c5fdd9ea65
HTML: Sort playlists alphabetically in watch page drop down (#4853)
Closes issue 4708
2024-08-24 20:50:46 +02:00
Samantaz Fox
2876ee0f9f
HTML: Fix XSS vulnerability in description/comments (#4852)
Before this PR, the comment/description content was not HTML escaped when 'parse_description()'
was called with a JSON object lacking the "commandRuns" entry.

Closes issue 4727
2024-08-24 20:50:05 +02:00
Samantaz Fox
0699e5fc27
YtAPI: Bump client versions (#4849)
This might help reducing the amount of playback errors.

No related issue
2024-08-24 20:47:01 +02:00
Samantaz Fox
15669acccf
SigHelper: Fix inverted time comparison in 'check_update' (#4845)
Closes issue 4840
2024-08-24 20:44:52 +02:00
Samantaz Fox
cd2daf4adb
Storyboards: Various fixes and code cleaning (#4153)
Closes issue 3441
2024-08-24 20:43:05 +02:00
syeopite
ccecc6d318
Fix lint errors introduced in #4146 and #4295 (#4876)
* Ameba: Fix Naming/VariableNames

Introduced in #4295

* Ameba: Fix Naming/PredicateName

Introduced in #4146
2024-08-24 18:11:11 +00:00
Samantaz Fox
3c6a662aaf
Search: Add support for Youtube URLs (#4146)
Closes issue 3300
2024-08-24 19:44:59 +02:00
Samantaz Fox
9e55799269
Channel: Render age restricted channels (#4295)
This PR:
 * gets thumbnail and channel name from the initial request
 * gets videos, shorts and streams via autogenerated channel playlists

Test Url: /channel/UCbfnHqxXs_K3kvaH-WlNlig

Closes issue 3513
2024-08-24 19:43:59 +02:00
Samantaz Fox
da70c9b7b0
Ameba: Miscellaneous fixes (#4807)
End of a series of PRs meant to improve code quality.

Related to issue 2231
2024-08-24 19:42:10 +02:00
Samantaz Fox
828da3c6ce
API: Proxy formatStreams URLs too (#4859)
The /api/v1/videos endpoint does not proxy the formatStreams URLs when
'local=true' is passed, whereas the adaptiveFormats URLs are correctly proxied.

The Web UI does proxy when clicking "Download" with 'fmt=18' for example, so
this is probably an oversight. This PR aims to fix that.

No related issue
2024-08-24 19:39:36 +02:00
Samantaz Fox
febf18cbf7
UI: Add search button to search bar (#4706)
Closes issue 529
2024-08-24 19:38:48 +02:00
Samantaz Fox
21ab5dc668
Storyboard: Revert cue timing "fix" 2024-08-22 00:29:15 +02:00
Samantaz Fox
b200ebfb6b
CSS: Remove extra space in default.css 2024-08-21 20:23:45 +00:00
syeopite
ecbea0b67b
Ameba: Fix Lint/ShadowingOuterLocalVar 2024-08-21 02:43:26 -07:00
syeopite
d1cd790388
Ameba: Fix Lint/RedundantStringCoercion 2024-08-21 02:43:26 -07:00
syeopite
f66068976e
Ameba: Fix Naming/PredicateName 2024-08-21 02:43:08 -07:00
syeopite
22b35c453e
Ameba: Fix Style/WhileTrue 2024-08-21 02:43:08 -07:00
Colin Leroy-Mira
c606465708 Proxify formatStreams URLs too 2024-08-19 09:37:24 +02:00
Samantaz Fox
85deea5aca
Search: Change smart search inhibitor to a backslash 2024-08-17 19:22:40 +02:00
Samantaz Fox
78c5ba93c7
Misc: Clean some code in UrlSanitizer 2024-08-17 19:22:40 +02:00
Samantaz Fox
31a80420ec
Search: Add URL search inhibition logic 2024-08-17 19:22:40 +02:00
Samantaz Fox
4c0b5c314d
Search: Add support for youtu.be and youtube.com URLs 2024-08-17 19:22:40 +02:00
Samantaz Fox
eb0f651812
Add a youtube URL sanitizer 2024-08-17 19:22:40 +02:00
Samantaz Fox
764965c441
Storyboards: Fix lint error 2024-08-17 12:20:53 +02:00
Samantaz Fox
b795bdf2a4
HTML: Sort playlists alphabetically in watch page drop down 2024-08-16 12:10:22 +02:00
Samantaz Fox
5b05f3bd14
Storyboards: Workarounds for videojs-vtt-thumbnails
The workarounds are as follow:
  * Unescape HTML entities
  * Always use 0:00:00.000 for cue start/end
2024-08-16 11:36:01 +02:00
Samantaz Fox
a335bc0814
Storyboards: Fix some small logic mistakes 2024-08-16 10:05:49 +02:00
Samantaz Fox
7b50388eaf
Storyboards: Fix broken first storyboard 2024-08-16 10:05:48 +02:00
Samantaz Fox
da3d58f03c
Storyboards: Cleanup and document code 2024-08-16 10:05:47 +02:00
Samantaz Fox
8327862697
Storyboards: Use replace the NamedTuple by a struct 2024-08-16 10:04:40 +02:00
Samantaz Fox
6878822c4d
Storyboards: Move parser to its own file 2024-08-16 10:02:52 +02:00
Samantaz Fox
0b28054f8a
videos: Fix XSS vulnerability in description/comments
Patch provided by e-mail, thanks to an anonymous user whose cats are named
Yoshi and Yasuo.

Comment is mine
2024-08-15 18:26:17 +02:00
Samantaz Fox
cc33d3f074
YtAPI: Also update User-Agent string 2024-08-15 18:14:29 +02:00
Samantaz Fox
acbb625866
YtAPI: Update clients to latest version 2024-08-15 12:57:36 +02:00
Samantaz Fox
466bfbb306
SigHelper: Fix inverted time comparison in 'check_update' 2024-08-14 21:43:37 +02:00
ChunkyProgrammer
e31053e812 Use dig to get properties
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
2024-08-13 15:09:16 -04:00
ChunkyProgrammer
96ade642fa Channel: Render age restricted channels 2024-08-13 15:09:16 -04:00
thansk
1ce2d10c50
fix: use ion icon for search icon 2024-05-20 14:17:30 +00:00
thansk
5abafb8296
fix: use a search icon instead of text 2024-05-20 11:49:56 +00:00
thansk
9cd2e93a2e
feat: allow submitting search with mouse 2024-05-19 11:46:55 +00:00
fkrueger
4a70164539 FK: let crystal tool format do its magic and replace the spaces before in-line comments with a random number of tabs, to get the lint check to work 2024-05-10 13:22:24 +02:00
fkrueger
1e3bbe2781 Merge remote-tracking branch 'origin/master' into fix-for-YT-playlist-import
Merging fix-for-YT-playlist-import with master from 2024-04-27
2024-04-27 18:37:08 +02:00
fkrueger
e5655008cc FK: I m new to crystal, ok? I made the error to trust the first official looking crystal (reports) documentation that showed up on google. perl-syntax for elsif definitely is preferrable to the basic BS of else if :-) 2024-01-07 12:32:00 +01:00
fkrueger
7775f46fc6 FK: cleaned up the "logic" and added a few tags for future reference 2024-01-07 12:25:57 +01:00
fkrueger
1e2d8623e2 FK: created a workaround for this 2+ months old problem import googles export / takeout of youtube playlists. not perfect (ie. things like "providing the takeout zip and all playlists get imported".. not yet supported). 2024-01-07 02:53:25 +01:00
25 changed files with 894 additions and 270 deletions

View File

@ -1,6 +1,7 @@
name: Build and release container
on:
workflow_dispatch:
push:
tags:
- "v*"
@ -46,9 +47,11 @@ jobs:
uses: docker/metadata-action@v5
with:
images: quay.io/invidious/invidious
flavor: |
latest=false
tags: |
type=semver,pattern={{version}}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=latest
labels: |
quay.expires-after=12w
@ -70,10 +73,11 @@ jobs:
with:
images: quay.io/invidious/invidious
flavor: |
latest=false
suffix=-arm64
tags: |
type=semver,pattern={{version}}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=latest
labels: |
quay.expires-after=12w

View File

@ -1,6 +1,189 @@
# CHANGELOG
## 2024-04-26
## v2.20240825.2 (2024-08-26)
This releases fixes the container tags pushed on quay.io.
Previously, the ARM64 build was released under the `latest` tag, instead of `latest-arm64`.
### Full list of pull requests merged since the last release (newest first)
CI: Fix docker container tags ([#4883], by @SamantazFox)
[#4877]: https://github.com/iv-org/invidious/pull/4877
## v2.20240825.1 (2024-08-25)
Add patch component to be [semver] compliant and make github actions happy.
[semver]: https://semver.org/
### Full list of pull requests merged since the last release (newest first)
Allow manual trigger of release-container build ([#4877], thanks @syeopite)
[#4877]: https://github.com/iv-org/invidious/pull/4877
## v2.20240825.0 (2024-08-25)
### New features & important changes
#### For users
* The search bar now has a button that you can click!
* Youtube URLs can be pasted directly in the search bar. Prepend search query with a
backslash (`\`) to disable that feature (useful if you need to search for a video whose
title contains some youtube URL).
* On the channel page the "streams" tab can be sorted by either: "newest", "oldest" or "popular"
* Lots of translations have been updated (thanks to our contributors on Weblate!)
* Videos embedded in local HTML files (e.g: a webpage saved from a blog) can now be played
#### For instance owners
* Invidious now has the ability to provide a `po_token` and `visitordata` to Youtube in order to
circumvent current Youtube restrictions.
* Invidious can use an (optional) external signature server like [inv_sig_helper]. Please note that
some videos can't be played without that signature server.
* The Helm charts were moved to a separate repo: https://github.com/iv-org/invidious-helm-chart
* We have changed how containers are released: the `latest` tag now tracks tagged releases, whereas
the `master` tag tracks the most recent commits of the `master` branch ("nightly" builds).
[inv_sig_helper]: https://github.com/iv-org/inv_sig_helper
#### For developpers
* The versions of Crystal that we test in CI/CD are now: `1.9.2`, `1.10.1`, `1.11.2`, `1.12.1`.
Please note that due to a bug in the `libxml` bindings (See [#4256]), versions prior to `1.10.0`
are not recommended to use.
* Thanks to @syeopite, the code is now [ameba] compliant.
* Ameba is part of our CI/CD pipeline, and its rules will be enforced in future PRs.
* The transcript code has been rewritten to permit transcripts as a feature rather than being
only a workaround for captions. Trancripts feature is coming soon!
* Various fixes regarding the logic interacting with Youtube
* The `sort_by` parameter can be used on the `/api/v1/channels/{id}/streams` endpoint. Accepted
values are: "newest", "oldest" and "popular"
[ameba]: https://github.com/crystal-ameba/ameba
[#4256]: https://github.com/iv-org/invidious/issues/4256
### Bugs fixed
#### User-side
* Channels: fixed broken "subscribers" and "views" counters
* Watch page: playback position is reset at the end of a video, so that the next time this video
is watched, it will start from the beginning rather than 15 seconds before the end
* Watch page: the items in the "add to playlist" drop down are now sorted alphabetically
* Videos: the "genre" URL is now always pointing to a valid webpage
* Playlists: Fixed `Could not parse N episodes` error on podcast playlists
* All external links should now have the [`rel`] attibute set to `noreferrer noopener` for
increased privacy.
* Preferences: Fixed the admin-only "modified source code" input being ignored
* Watch/channel pages: use the full image URL in `og:image` and `twitter:image` meta tags
[`rel`]: https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel
#### API
* fixed the `local` parameter not applying to `formatStreams` on `/api/v1/videos/{id}`
* fixed an `Index out of bounds` error hapenning when a playlist had no videos
* fixed duplicated query parameters in proxied video URLs
* Return actual video height/width/fps rather than hard coded values
* Fixed the `/api/v1/popular` endpoint not returning a proper error code/message when the
popular page/endpoint are disabled.
### Full list of pull requests merged since the last release (newest first)
* HTML: Sort playlists alphabetically in watch page drop down ([#4853], by @SamantazFox)
* Videos: Fix XSS vulnerability in description/comments ([#4852], thanks _anonymous_)
* YtAPI: Bump client versions ([#4849], by @SamantazFox)
* SigHelper: Fix inverted time comparison in 'check_update' ([#4845], by @SamantazFox)
* Storyboards: Various fixes and code cleaning ([#4153], by SamantazFox)
* Fix lint errors introduced in #4146 and #4295 ([#4876], thanks @syeopite)
* Search: Add support for Youtube URLs ([#4146], by @SamantazFox)
* Channel: Render age restricted channels ([#4295], thanks @ChunkyProgrammer)
* Ameba: Miscellaneous fixes ([#4807], thanks @syeopite)
* API: Proxy formatStreams URLs too ([#4859], thanks @colinleroy)
* UI: Add search button to search bar ([#4706], thanks @thansk)
* Add ability to set po_token and visitordata ID ([#4789], thanks @unixfox)
* Add support for an external signature server ([#4772], by @SamantazFox)
* Ameba: Fix Naming/VariableNames ([#4790], thanks @syeopite)
* Translations update from Hosted Weblate ([#4659])
* Ameba: Fix Lint/UselessAssign ([#4795], thanks @syeopite)
* HTML: Add rel="noreferrer noopener" to external links ([#4667], thanks @ulmemxpoc)
* Remove unused methods in Invidious::LogHandler ([#4812], thanks @syeopite)
* Ameba: Fix Lint/NotNilAfterNoBang ([#4796], thanks @syeopite)
* Ameba: Fix unused argument Lint warnings ([#4805], thanks @syeopite)
* Ameba: i18next.cr fixes ([#4806], thanks @syeopite)
* Ameba: Disable rules ([#4792], thanks @syeopite)
* Channel: parse subscriber count and channel banner ([#4785], thanks @ChunkyProgrammer)
* Player: Fix playback position of already watched videos ([#4731], thanks @Fijxu)
* Videos: Fix genre url being unusable ([#4717], thanks @meatball133)
* API: Fix out of bound error on empty playlists ([#4696], thanks @Fijxu)
* Handle playlists cataloged as Podcast ([#4695], thanks @Fijxu)
* API: Fix duplicated query parameters in proxied video URLs ([#4587], thanks @absidue)
* API: Return actual stream height, width and fps ([#4586], thanks @absidue)
* Preferences: Fix handling of modified source code URL ([#4437], thanks @nooptek)
* API: Fix URL for vtt subtitles ([#4221], thanks @karelrooted)
* Channels: Add sort options to streams ([#4224], thanks @src-tinkerer)
* API: Fix error code for disabled popular endpoint ([#4296], thanks @iBicha)
* Allow embedding videos in local HTML files ([#4450], thanks @tomasz1986)
* CI: Bump Crystal version matrix ([#4654], by @SamantazFox)
* YtAPI: Remove API keys like official clients ([#4655], by @SamantazFox)
* HTML: Use full URL in the og:image property ([#4675], thanks @Fijxu)
* Rewrite transcript logic to be more generic ([#4747], thanks @syeopite)
* CI: Run Ameba ([#4753], thanks @syeopite)
* CI: Add release based containers ([#4763], thanks @syeopite)
* move helm chart to a dedicated github repository ([#4711], thanks @unixfox)
[#4146]: https://github.com/iv-org/invidious/pull/4146
[#4153]: https://github.com/iv-org/invidious/pull/4153
[#4221]: https://github.com/iv-org/invidious/pull/4221
[#4224]: https://github.com/iv-org/invidious/pull/4224
[#4295]: https://github.com/iv-org/invidious/pull/4295
[#4296]: https://github.com/iv-org/invidious/pull/4296
[#4437]: https://github.com/iv-org/invidious/pull/4437
[#4450]: https://github.com/iv-org/invidious/pull/4450
[#4586]: https://github.com/iv-org/invidious/pull/4586
[#4587]: https://github.com/iv-org/invidious/pull/4587
[#4654]: https://github.com/iv-org/invidious/pull/4654
[#4655]: https://github.com/iv-org/invidious/pull/4655
[#4659]: https://github.com/iv-org/invidious/pull/4659
[#4667]: https://github.com/iv-org/invidious/pull/4667
[#4675]: https://github.com/iv-org/invidious/pull/4675
[#4695]: https://github.com/iv-org/invidious/pull/4695
[#4696]: https://github.com/iv-org/invidious/pull/4696
[#4706]: https://github.com/iv-org/invidious/pull/4706
[#4711]: https://github.com/iv-org/invidious/pull/4711
[#4717]: https://github.com/iv-org/invidious/pull/4717
[#4731]: https://github.com/iv-org/invidious/pull/4731
[#4747]: https://github.com/iv-org/invidious/pull/4747
[#4753]: https://github.com/iv-org/invidious/pull/4753
[#4763]: https://github.com/iv-org/invidious/pull/4763
[#4772]: https://github.com/iv-org/invidious/pull/4772
[#4785]: https://github.com/iv-org/invidious/pull/4785
[#4789]: https://github.com/iv-org/invidious/pull/4789
[#4790]: https://github.com/iv-org/invidious/pull/4790
[#4792]: https://github.com/iv-org/invidious/pull/4792
[#4795]: https://github.com/iv-org/invidious/pull/4795
[#4796]: https://github.com/iv-org/invidious/pull/4796
[#4805]: https://github.com/iv-org/invidious/pull/4805
[#4806]: https://github.com/iv-org/invidious/pull/4806
[#4807]: https://github.com/iv-org/invidious/pull/4807
[#4812]: https://github.com/iv-org/invidious/pull/4812
[#4845]: https://github.com/iv-org/invidious/pull/4845
[#4849]: https://github.com/iv-org/invidious/pull/4849
[#4852]: https://github.com/iv-org/invidious/pull/4852
[#4853]: https://github.com/iv-org/invidious/pull/4853
[#4859]: https://github.com/iv-org/invidious/pull/4859
[#4876]: https://github.com/iv-org/invidious/pull/4876
## v2.20240427 (2024-04-27)
Major bug fixes:
* Videos: Use android test suite client (#4650, thanks @SamantazFox)

View File

@ -278,7 +278,14 @@ div.thumbnail > .bottom-right-overlay {
display: inline;
}
.searchbar .pure-form fieldset { padding: 0; }
.searchbar .pure-form {
display: flex;
}
.searchbar .pure-form fieldset {
padding: 0;
flex: 1;
}
.searchbar input[type="search"] {
width: 100%;
@ -310,6 +317,16 @@ input[type="search"]::-webkit-search-cancel-button {
background-size: 14px;
}
.searchbar #searchbutton {
border: none;
background: none;
margin-top: 0;
}
.searchbar #searchbutton:hover {
color: rgb(0, 182, 240);
}
.user-field {
display: flex;
flex-direction: row;

View File

@ -15,7 +15,8 @@ record AboutChannel,
allowed_regions : Array(String),
tabs : Array(String),
tags : Array(String),
verified : Bool
verified : Bool,
is_age_gated : Bool
def get_about_info(ucid, locale) : AboutChannel
begin
@ -45,46 +46,102 @@ def get_about_info(ucid, locale) : AboutChannel
end
tags = [] of String
tab_names = [] of String
total_views = 0_i64
joined = Time.unix(0)
if auto_generated
author = initdata["header"]["interactiveTabbedHeaderRenderer"]["title"]["simpleText"].as_s
author_url = initdata["microformat"]["microformatDataRenderer"]["urlCanonical"].as_s
author_thumbnail = initdata["header"]["interactiveTabbedHeaderRenderer"]["boxArt"]["thumbnails"][0]["url"].as_s
# Raises a KeyError on failure.
banners = initdata["header"]["interactiveTabbedHeaderRenderer"]?.try &.["banner"]?.try &.["thumbnails"]?
banner = banners.try &.[-1]?.try &.["url"].as_s?
description_base_node = initdata["header"]["interactiveTabbedHeaderRenderer"]["description"]
# some channels have the description in a simpleText
# ex: https://www.youtube.com/channel/UCQvWX73GQygcwXOTSf_VDVg/
description_node = description_base_node.dig?("simpleText") || description_base_node
tags = initdata.dig?("header", "interactiveTabbedHeaderRenderer", "badges")
.try &.as_a.map(&.["metadataBadgeRenderer"]["label"].as_s) || [] of String
if age_gate_renderer = initdata.dig?("contents", "twoColumnBrowseResultsRenderer", "tabs", 0, "tabRenderer", "content", "sectionListRenderer", "contents", 0, "channelAgeGateRenderer")
description_node = nil
author = age_gate_renderer["channelTitle"].as_s
ucid = initdata.dig("responseContext", "serviceTrackingParams", 0, "params", 0, "value").as_s
author_url = "https://www.youtube.com/channel/#{ucid}"
author_thumbnail = age_gate_renderer.dig("avatar", "thumbnails", 0, "url").as_s
banner = nil
is_family_friendly = false
is_age_gated = true
tab_names = ["videos", "shorts", "streams"]
auto_generated = false
else
author = initdata["metadata"]["channelMetadataRenderer"]["title"].as_s
author_url = initdata["metadata"]["channelMetadataRenderer"]["channelUrl"].as_s
author_thumbnail = initdata["metadata"]["channelMetadataRenderer"]["avatar"]["thumbnails"][0]["url"].as_s
author_verified = has_verified_badge?(initdata.dig?("header", "c4TabbedHeaderRenderer", "badges"))
if auto_generated
author = initdata["header"]["interactiveTabbedHeaderRenderer"]["title"]["simpleText"].as_s
author_url = initdata["microformat"]["microformatDataRenderer"]["urlCanonical"].as_s
author_thumbnail = initdata["header"]["interactiveTabbedHeaderRenderer"]["boxArt"]["thumbnails"][0]["url"].as_s
ucid = initdata["metadata"]["channelMetadataRenderer"]["externalId"].as_s
# Raises a KeyError on failure.
banners = initdata["header"]["interactiveTabbedHeaderRenderer"]?.try &.["banner"]?.try &.["thumbnails"]?
banner = banners.try &.[-1]?.try &.["url"].as_s?
# Raises a KeyError on failure.
banners = initdata["header"]["c4TabbedHeaderRenderer"]?.try &.["banner"]?.try &.["thumbnails"]?
banners ||= initdata.dig?("header", "pageHeaderRenderer", "content", "pageHeaderViewModel", "banner", "imageBannerViewModel", "image", "sources")
banner = banners.try &.[-1]?.try &.["url"].as_s?
description_base_node = initdata["header"]["interactiveTabbedHeaderRenderer"]["description"]
# some channels have the description in a simpleText
# ex: https://www.youtube.com/channel/UCQvWX73GQygcwXOTSf_VDVg/
description_node = description_base_node.dig?("simpleText") || description_base_node
# if banner.includes? "channels/c4/default_banner"
# banner = nil
# end
tags = initdata.dig?("header", "interactiveTabbedHeaderRenderer", "badges")
.try &.as_a.map(&.["metadataBadgeRenderer"]["label"].as_s) || [] of String
else
author = initdata["metadata"]["channelMetadataRenderer"]["title"].as_s
author_url = initdata["metadata"]["channelMetadataRenderer"]["channelUrl"].as_s
author_thumbnail = initdata["metadata"]["channelMetadataRenderer"]["avatar"]["thumbnails"][0]["url"].as_s
author_verified = has_verified_badge?(initdata.dig?("header", "c4TabbedHeaderRenderer", "badges"))
description_node = initdata["metadata"]["channelMetadataRenderer"]?.try &.["description"]?
tags = initdata.dig?("microformat", "microformatDataRenderer", "tags").try &.as_a.map(&.as_s) || [] of String
ucid = initdata["metadata"]["channelMetadataRenderer"]["externalId"].as_s
# Raises a KeyError on failure.
banners = initdata["header"]["c4TabbedHeaderRenderer"]?.try &.["banner"]?.try &.["thumbnails"]?
banners ||= initdata.dig?("header", "pageHeaderRenderer", "content", "pageHeaderViewModel", "banner", "imageBannerViewModel", "image", "sources")
banner = banners.try &.[-1]?.try &.["url"].as_s?
# if banner.includes? "channels/c4/default_banner"
# banner = nil
# end
description_node = initdata["metadata"]["channelMetadataRenderer"]?.try &.["description"]?
tags = initdata.dig?("microformat", "microformatDataRenderer", "tags").try &.as_a.map(&.as_s) || [] of String
end
is_family_friendly = initdata["microformat"]["microformatDataRenderer"]["familySafe"].as_bool
if tabs_json = initdata["contents"]["twoColumnBrowseResultsRenderer"]["tabs"]?
# Get the name of the tabs available on this channel
tab_names = tabs_json.as_a.compact_map do |entry|
name = entry.dig?("tabRenderer", "title").try &.as_s.downcase
# This is a small fix to not add extra code on the HTML side
# I.e, the URL for the "live" tab is .../streams, so use "streams"
# everywhere for the sake of simplicity
(name == "live") ? "streams" : name
end
# Get the currently active tab ("About")
about_tab = extract_selected_tab(tabs_json)
# Try to find the about metadata section
channel_about_meta = about_tab.dig?(
"content",
"sectionListRenderer", "contents", 0,
"itemSectionRenderer", "contents", 0,
"channelAboutFullMetadataRenderer"
)
if !channel_about_meta.nil?
total_views = channel_about_meta.dig?("viewCountText", "simpleText").try &.as_s.gsub(/\D/, "").to_i64? || 0_i64
# The joined text is split to several sub strings. The reduce joins those strings before parsing the date.
joined = extract_text(channel_about_meta["joinedDateText"]?)
.try { |text| Time.parse(text, "Joined %b %-d, %Y", Time::Location.local) } || Time.unix(0)
# Normal Auto-generated channels
# https://support.google.com/youtube/answer/2579942
# For auto-generated channels, channel_about_meta only has
# ["description"]["simpleText"] and ["primaryLinks"][0]["title"]["simpleText"]
auto_generated = (
(channel_about_meta["primaryLinks"]?.try &.size) == 1 && \
extract_text(channel_about_meta.dig?("primaryLinks", 0, "title")) == "Auto-generated by YouTube" ||
channel_about_meta.dig?("links", 0, "channelExternalLinkViewModel", "title", "content").try &.as_s == "Auto-generated by YouTube"
)
end
end
end
is_family_friendly = initdata["microformat"]["microformatDataRenderer"]["familySafe"].as_bool
allowed_regions = initdata
.dig?("microformat", "microformatDataRenderer", "availableCountries")
.try &.as_a.map(&.as_s) || [] of String
@ -102,52 +159,6 @@ def get_about_info(ucid, locale) : AboutChannel
end
end
total_views = 0_i64
joined = Time.unix(0)
tab_names = [] of String
if tabs_json = initdata["contents"]["twoColumnBrowseResultsRenderer"]["tabs"]?
# Get the name of the tabs available on this channel
tab_names = tabs_json.as_a.compact_map do |entry|
name = entry.dig?("tabRenderer", "title").try &.as_s.downcase
# This is a small fix to not add extra code on the HTML side
# I.e, the URL for the "live" tab is .../streams, so use "streams"
# everywhere for the sake of simplicity
(name == "live") ? "streams" : name
end
# Get the currently active tab ("About")
about_tab = extract_selected_tab(tabs_json)
# Try to find the about metadata section
channel_about_meta = about_tab.dig?(
"content",
"sectionListRenderer", "contents", 0,
"itemSectionRenderer", "contents", 0,
"channelAboutFullMetadataRenderer"
)
if !channel_about_meta.nil?
total_views = channel_about_meta.dig?("viewCountText", "simpleText").try &.as_s.gsub(/\D/, "").to_i64? || 0_i64
# The joined text is split to several sub strings. The reduce joins those strings before parsing the date.
joined = extract_text(channel_about_meta["joinedDateText"]?)
.try { |text| Time.parse(text, "Joined %b %-d, %Y", Time::Location.local) } || Time.unix(0)
# Normal Auto-generated channels
# https://support.google.com/youtube/answer/2579942
# For auto-generated channels, channel_about_meta only has
# ["description"]["simpleText"] and ["primaryLinks"][0]["title"]["simpleText"]
auto_generated = (
(channel_about_meta["primaryLinks"]?.try &.size) == 1 && \
extract_text(channel_about_meta.dig?("primaryLinks", 0, "title")) == "Auto-generated by YouTube" ||
channel_about_meta.dig?("links", 0, "channelExternalLinkViewModel", "title", "content").try &.as_s == "Auto-generated by YouTube"
)
end
end
sub_count = 0
if (metadata_rows = initdata.dig?("header", "pageHeaderRenderer", "content", "pageHeaderViewModel", "metadata", "contentMetadataViewModel", "metadataRows").try &.as_a)
@ -177,6 +188,7 @@ def get_about_info(ucid, locale) : AboutChannel
tabs: tab_names,
tags: tags,
verified: author_verified || false,
is_age_gated: is_age_gated || false,
)
end

View File

@ -140,6 +140,7 @@ module Invidious::Database::Playlists
request = <<-SQL
SELECT id,title FROM playlists
WHERE author = $1 AND id LIKE 'IV%'
ORDER BY title
SQL
PG_DB.query_all(request, email, as: {String, String})

View File

@ -90,7 +90,7 @@ struct SearchVideo
json.field "lengthSeconds", self.length_seconds
json.field "liveNow", self.live_now
json.field "premium", self.premium
json.field "isUpcoming", self.is_upcoming
json.field "isUpcoming", self.upcoming?
if self.premiere_timestamp
json.field "premiereTimestamp", self.premiere_timestamp.try &.to_unix
@ -109,7 +109,7 @@ struct SearchVideo
to_json(nil, json)
end
def is_upcoming
def upcoming?
premiere_timestamp ? true : false
end
end

View File

@ -10,10 +10,8 @@ class Invidious::DecryptFunction
end
def check_update
now = Time.utc
# If we have updated in the last 5 minutes, do nothing
return if (now - @last_update) > 5.minutes
return if (Time.utc - @last_update) < 5.minutes
# Get the amount of time elapsed since when the player was updated, in the
# event where multiple invidious processes are run in parallel.

View File

@ -63,7 +63,7 @@ module Invidious::JSONify::APIv1
json.field "isListed", video.is_listed
json.field "liveNow", video.live_now
json.field "isPostLiveDvr", video.post_live_dvr
json.field "isUpcoming", video.is_upcoming
json.field "isUpcoming", video.upcoming?
if video.premiere_timestamp
json.field "premiereTimestamp", video.premiere_timestamp.try &.to_unix
@ -109,7 +109,7 @@ module Invidious::JSONify::APIv1
# On livestreams, it's not present, so always fall back to the
# current unix timestamp (up to mS precision) for compatibility.
last_modified = fmt["lastModified"]?
last_modified ||= "#{Time.utc.to_unix_ms.to_s}000"
last_modified ||= "#{Time.utc.to_unix_ms}000"
json.field "lmt", last_modified
json.field "projectionType", fmt["projectionType"]
@ -162,7 +162,13 @@ module Invidious::JSONify::APIv1
json.array do
video.fmt_stream.each do |fmt|
json.object do
json.field "url", fmt["url"]
if proxy
json.field "url", Invidious::HttpServer::Utils.proxy_video_url(
fmt["url"].to_s, absolute: true
)
else
json.field "url", fmt["url"]
end
json.field "itag", fmt["itag"].as_i.to_s
json.field "type", fmt["mimeType"]
json.field "quality", fmt["quality"]
@ -271,17 +277,17 @@ module Invidious::JSONify::APIv1
def storyboards(json, id, storyboards)
json.array do
storyboards.each do |storyboard|
storyboards.each do |sb|
json.object do
json.field "url", "/api/v1/storyboards/#{id}?width=#{storyboard[:width]}&height=#{storyboard[:height]}"
json.field "templateUrl", storyboard[:url]
json.field "width", storyboard[:width]
json.field "height", storyboard[:height]
json.field "count", storyboard[:count]
json.field "interval", storyboard[:interval]
json.field "storyboardWidth", storyboard[:storyboard_width]
json.field "storyboardHeight", storyboard[:storyboard_height]
json.field "storyboardCount", storyboard[:storyboard_count]
json.field "url", "/api/v1/storyboards/#{id}?width=#{sb.width}&height=#{sb.height}"
json.field "templateUrl", sb.url.to_s
json.field "width", sb.width
json.field "height", sb.height
json.field "count", sb.count
json.field "interval", sb.interval
json.field "storyboardWidth", sb.columns
json.field "storyboardHeight", sb.rows
json.field "storyboardCount", sb.images_count
end
end
end

View File

@ -46,8 +46,14 @@ struct PlaylistVideo
XML.build { |xml| to_xml(xml) }
end
def to_json(locale : String?, json : JSON::Builder)
to_json(json)
end
def to_json(json : JSON::Builder, index : Int32? = nil)
json.object do
json.field "type", "video"
json.field "title", self.title
json.field "videoId", self.id
@ -67,6 +73,7 @@ struct PlaylistVideo
end
json.field "lengthSeconds", self.length_seconds
json.field "liveNow", self.live_now
end
end

View File

@ -27,10 +27,21 @@ module Invidious::Routes::API::V1::Channels
# Retrieve "sort by" setting from URL parameters
sort_by = env.params.query["sort_by"]?.try &.downcase || "newest"
begin
videos, _ = Channel::Tabs.get_videos(channel, sort_by: sort_by)
rescue ex
return error_json(500, ex)
if channel.is_age_gated
begin
playlist = get_playlist(channel.ucid.sub("UC", "UULF"))
videos = get_playlist_videos(playlist, offset: 0)
rescue ex : InfoException
# playlist doesnt exist.
videos = [] of PlaylistVideo
end
next_continuation = nil
else
begin
videos, _ = Channel::Tabs.get_videos(channel, sort_by: sort_by)
rescue ex
return error_json(500, ex)
end
end
JSON.build do |json|
@ -84,6 +95,7 @@ module Invidious::Routes::API::V1::Channels
json.field "joined", channel.joined.to_unix
json.field "autoGenerated", channel.auto_generated
json.field "ageGated", channel.is_age_gated
json.field "isFamilyFriendly", channel.is_family_friendly
json.field "description", html_to_content(channel.description_html)
json.field "descriptionHtml", channel.description_html
@ -142,12 +154,23 @@ module Invidious::Routes::API::V1::Channels
sort_by = env.params.query["sort_by"]?.try &.downcase || "newest"
continuation = env.params.query["continuation"]?
begin
videos, next_continuation = Channel::Tabs.get_60_videos(
channel, continuation: continuation, sort_by: sort_by
)
rescue ex
return error_json(500, ex)
if channel.is_age_gated
begin
playlist = get_playlist(channel.ucid.sub("UC", "UULF"))
videos = get_playlist_videos(playlist, offset: 0)
rescue ex : InfoException
# playlist doesnt exist.
videos = [] of PlaylistVideo
end
next_continuation = nil
else
begin
videos, next_continuation = Channel::Tabs.get_60_videos(
channel, continuation: continuation, sort_by: sort_by
)
rescue ex
return error_json(500, ex)
end
end
return JSON.build do |json|
@ -176,12 +199,23 @@ module Invidious::Routes::API::V1::Channels
# Retrieve continuation from URL parameters
continuation = env.params.query["continuation"]?
begin
videos, next_continuation = Channel::Tabs.get_shorts(
channel, continuation: continuation
)
rescue ex
return error_json(500, ex)
if channel.is_age_gated
begin
playlist = get_playlist(channel.ucid.sub("UC", "UUSH"))
videos = get_playlist_videos(playlist, offset: 0)
rescue ex : InfoException
# playlist doesnt exist.
videos = [] of PlaylistVideo
end
next_continuation = nil
else
begin
videos, next_continuation = Channel::Tabs.get_shorts(
channel, continuation: continuation
)
rescue ex
return error_json(500, ex)
end
end
return JSON.build do |json|
@ -211,12 +245,23 @@ module Invidious::Routes::API::V1::Channels
sort_by = env.params.query["sort_by"]?.try &.downcase || "newest"
continuation = env.params.query["continuation"]?
begin
videos, next_continuation = Channel::Tabs.get_60_livestreams(
channel, continuation: continuation, sort_by: sort_by
)
rescue ex
return error_json(500, ex)
if channel.is_age_gated
begin
playlist = get_playlist(channel.ucid.sub("UC", "UULV"))
videos = get_playlist_videos(playlist, offset: 0)
rescue ex : InfoException
# playlist doesnt exist.
videos = [] of PlaylistVideo
end
next_continuation = nil
else
begin
videos, next_continuation = Channel::Tabs.get_60_livestreams(
channel, continuation: continuation, sort_by: sort_by
)
rescue ex
return error_json(500, ex)
end
end
return JSON.build do |json|

View File

@ -1,3 +1,5 @@
require "html"
module Invidious::Routes::API::V1::Videos
def self.videos(env)
locale = env.get("preferences").as(Preferences).locale
@ -116,7 +118,7 @@ module Invidious::Routes::API::V1::Videos
else
caption_xml = XML.parse(caption_xml)
webvtt = WebVTT.build(settings_field) do |webvtt|
webvtt = WebVTT.build(settings_field) do |builder|
caption_nodes = caption_xml.xpath_nodes("//transcript/text")
caption_nodes.each_with_index do |node, i|
start_time = node["start"].to_f.seconds
@ -136,7 +138,7 @@ module Invidious::Routes::API::V1::Videos
text = "<v #{md["name"]}>#{md["text"]}</v>"
end
webvtt.cue(start_time, end_time, text)
builder.cue(start_time, end_time, text)
end
end
end
@ -187,15 +189,14 @@ module Invidious::Routes::API::V1::Videos
haltf env, 500
end
storyboards = video.storyboards
width = env.params.query["width"]?
height = env.params.query["height"]?
width = env.params.query["width"]?.try &.to_i
height = env.params.query["height"]?.try &.to_i
if !width && !height
response = JSON.build do |json|
json.object do
json.field "storyboards" do
Invidious::JSONify::APIv1.storyboards(json, id, storyboards)
Invidious::JSONify::APIv1.storyboards(json, id, video.storyboards)
end
end
end
@ -205,35 +206,48 @@ module Invidious::Routes::API::V1::Videos
env.response.content_type = "text/vtt"
storyboard = storyboards.select { |sb| width == "#{sb[:width]}" || height == "#{sb[:height]}" }
# Select a storyboard matching the user's provided width/height
storyboard = video.storyboards.select { |x| x.width == width || x.height == height }
haltf env, 404 if storyboard.empty?
if storyboard.empty?
haltf env, 404
else
storyboard = storyboard[0]
end
# Alias variable, to make the code below esaier to read
sb = storyboard[0]
WebVTT.build do |vtt|
start_time = 0.milliseconds
end_time = storyboard[:interval].milliseconds
# Some base URL segments that we'll use to craft the final URLs
work_url = sb.proxied_url.dup
template_path = sb.proxied_url.path
storyboard[:storyboard_count].times do |i|
url = storyboard[:url]
authority = /(i\d?).ytimg.com/.match!(url)[1]?
url = url.gsub("$M", i).gsub(%r(https://i\d?.ytimg.com/sb/), "")
url = "#{HOST_URL}/sb/#{authority}/#{url}"
# Initialize cue timing variables
# NOTE: videojs-vtt-thumbnails gets lost when the cue times don't overlap
# (i.e: if cue[n] end time is 1:06:25.000, cue[n+1] start time should be 1:06:25.000)
time_delta = sb.interval.milliseconds
start_time = 0.milliseconds
end_time = time_delta
storyboard[:storyboard_height].times do |j|
storyboard[:storyboard_width].times do |k|
current_cue_url = "#{url}#xywh=#{storyboard[:width] * k},#{storyboard[:height] * j},#{storyboard[:width] - 2},#{storyboard[:height]}"
vtt.cue(start_time, end_time, current_cue_url)
# Build a VTT file for VideoJS-vtt plugin
vtt_file = WebVTT.build do |vtt|
sb.images_count.times do |i|
# Replace the variable component part of the path
work_url.path = template_path.sub("$M", i)
start_time += storyboard[:interval].milliseconds
end_time += storyboard[:interval].milliseconds
sb.rows.times do |j|
sb.columns.times do |k|
# The URL fragment represents the offset of the thumbnail inside the storyboard image
work_url.fragment = "xywh=#{sb.width * k},#{sb.height * j},#{sb.width - 2},#{sb.height}"
vtt.cue(start_time, end_time, work_url.to_s)
start_time += time_delta
end_time += time_delta
end
end
end
end
# videojs-vtt-thumbnails is not compliant to the VTT specification, it
# doesn't unescape the HTML entities, so we have to do it here:
# TODO: remove this when we migrate to VideoJS 8
return HTML.unescape(vtt_file)
end
def self.annotations(env)

View File

@ -36,12 +36,24 @@ module Invidious::Routes::Channels
items = items.select(SearchPlaylist)
items.each(&.author = "")
else
sort_options = {"newest", "oldest", "popular"}
# Fetch items and continuation token
items, next_continuation = Channel::Tabs.get_videos(
channel, continuation: continuation, sort_by: (sort_by || "newest")
)
if channel.is_age_gated
sort_by = ""
sort_options = [] of String
begin
playlist = get_playlist(channel.ucid.sub("UC", "UULF"))
items = get_playlist_videos(playlist, offset: 0)
rescue ex : InfoException
# playlist doesnt exist.
items = [] of PlaylistVideo
end
next_continuation = nil
else
sort_options = {"newest", "oldest", "popular"}
items, next_continuation = Channel::Tabs.get_videos(
channel, continuation: continuation, sort_by: (sort_by || "newest")
)
end
end
selected_tab = Frontend::ChannelPage::TabsAvailable::Videos
@ -58,14 +70,27 @@ module Invidious::Routes::Channels
return env.redirect "/channel/#{channel.ucid}"
end
# TODO: support sort option for shorts
sort_by = ""
sort_options = [] of String
if channel.is_age_gated
sort_by = ""
sort_options = [] of String
begin
playlist = get_playlist(channel.ucid.sub("UC", "UUSH"))
items = get_playlist_videos(playlist, offset: 0)
rescue ex : InfoException
# playlist doesnt exist.
items = [] of PlaylistVideo
end
next_continuation = nil
else
# TODO: support sort option for shorts
sort_by = ""
sort_options = [] of String
# Fetch items and continuation token
items, next_continuation = Channel::Tabs.get_shorts(
channel, continuation: continuation
)
# Fetch items and continuation token
items, next_continuation = Channel::Tabs.get_shorts(
channel, continuation: continuation
)
end
selected_tab = Frontend::ChannelPage::TabsAvailable::Shorts
templated "channel"
@ -81,13 +106,26 @@ module Invidious::Routes::Channels
return env.redirect "/channel/#{channel.ucid}"
end
sort_by = env.params.query["sort_by"]?.try &.downcase || "newest"
sort_options = {"newest", "oldest", "popular"}
if channel.is_age_gated
sort_by = ""
sort_options = [] of String
begin
playlist = get_playlist(channel.ucid.sub("UC", "UULV"))
items = get_playlist_videos(playlist, offset: 0)
rescue ex : InfoException
# playlist doesnt exist.
items = [] of PlaylistVideo
end
next_continuation = nil
else
sort_by = env.params.query["sort_by"]?.try &.downcase || "newest"
sort_options = {"newest", "oldest", "popular"}
# Fetch items and continuation token
items, next_continuation = Channel::Tabs.get_60_livestreams(
channel, continuation: continuation, sort_by: sort_by
)
# Fetch items and continuation token
items, next_continuation = Channel::Tabs.get_60_livestreams(
channel, continuation: continuation, sort_by: sort_by
)
end
selected_tab = Frontend::ChannelPage::TabsAvailable::Streams
templated "channel"

View File

@ -51,6 +51,12 @@ module Invidious::Routes::Search
else
user = env.get? "user"
# An URL was copy/pasted in the search box.
# Redirect the user to the appropriate page.
if query.url?
return env.redirect UrlSanitizer.process(query.text).to_s
end
begin
items = query.process
rescue ex : ChannelSearchException

View File

@ -131,7 +131,7 @@ module Invidious::Routes::VideoPlayback
end
# TODO: Record bytes written so we can restart after a chunk fails
while true
loop do
if !range_end && content_length
range_end = content_length
end

View File

@ -20,6 +20,9 @@ module Invidious::Search
property region : String?
property channel : String = ""
# Flag that indicates if the smart search features have been disabled.
@inhibit_ssf : Bool = false
# Return true if @raw_query is either `nil` or empty
private def empty_raw_query?
return @raw_query.empty?
@ -48,10 +51,18 @@ module Invidious::Search
)
# Get the raw search query string (common to all search types). In
# Regular search mode, also look for the `search_query` URL parameter
if @type.regular?
@raw_query = params["q"]? || params["search_query"]? || ""
else
@raw_query = params["q"]? || ""
_raw_query = params["q"]?
_raw_query ||= params["search_query"]? if @type.regular?
_raw_query ||= ""
# Remove surrounding whitespaces. Mostly useful for copy/pasted URLs.
@raw_query = _raw_query.strip
# Check for smart features (ex: URL search) inhibitor (backslash).
# If inhibitor is present, remove it.
if @raw_query.starts_with?('\\')
@inhibit_ssf = true
@raw_query = @raw_query[1..]
end
# Get the page number (also common to all search types)
@ -85,7 +96,7 @@ module Invidious::Search
@filters = Filters.from_iv_params(params)
@channel = params["channel"]? || ""
if @filters.default? && @raw_query.includes?(':')
if @filters.default? && @raw_query.index(/\w:\w/)
# Parse legacy filters from query
@filters, @channel, @query, subs = Filters.from_legacy_filters(@raw_query)
else
@ -136,5 +147,22 @@ module Invidious::Search
return params
end
# Checks if the query is a standalone URL
def url? : Bool
# If the smart features have been inhibited, don't go further.
return false if @inhibit_ssf
# Only supported in regular search mode
return false if !@type.regular?
# If filters are present, that's a regular search
return false if !@filters.default?
# Simple heuristics: domain name
return @raw_query.starts_with?(
/(https?:\/\/)?(www\.)?(m\.)?youtu(\.be|be\.com)\//
)
end
end
end

View File

@ -30,21 +30,75 @@ struct Invidious::User
return subscriptions
end
def parse_playlist_export_csv(user : User, raw_input : String)
def parse_playlist_export_csv(user : User, raw_input : String, filename : String)
LOGGER.trace("parse_playlist_export_csv: 01 raw_input '#{raw_input}'\n")
raw_head = "" # playlists.csv - info-line for a given playlist in the google export, that's copied above the actual playlist-infos and separated by an empty line from
raw_body = "" # the actual playlist content, ie. a list of videos
# remove superfluous \n instances in front of and after any non-\n text.. so .. a trim?
raw_input = raw_input.strip('\n')
# Split the input into head and body content
raw_head, raw_body = raw_input.strip('\n').split("\n\n", limit: 2, remove_empty: true)
# Create the playlist from the head content
csv_head = CSV.new(raw_head.strip('\n'), headers: true)
csv_head.next
title = csv_head[4]
description = csv_head[5]
visibility = csv_head[6]
if visibility.compare("Public", case_insensitive: true) == 0
privacy = PlaylistPrivacy::Public
tmp = raw_input.split("\n\n", limit: 2, remove_empty: true)
if tmp.size > 1
raw_head = tmp[0]
raw_body = tmp[1]
else
privacy = PlaylistPrivacy::Private
raw_body = tmp[0]
end
LOGGER.trace("parse_playlist_export_csv: 02 raw_head '#{raw_head}' -----\nraw_body '#{raw_body}'\n")
# TODO create an import-feature (elsewhere), which works for the original google export format, ie. the playlists/ subdirectory content as a ZIP file.. or a complete youtube music export file, but this goes way beyond the scope of a playlist import.
# defaults
title = "title not set" # TODO i8n
description = "from #{filename}" # TODO i8n
visibility = "Private"
privacy = PlaylistPrivacy::Private
if (raw_head != "")
# ## XXX for documentation of current file format of playlists.csv (today it's 2024-01-07):
# 0 1 2 3 4 5 6 7 8 9 10 11
# Playlist ID,Add new videos to top,Playlist image 1 Create timestamp,Playlist image 1 URL,Playlist image 1 Height,Playlist image 1 Width,Playlist title (original),Playlist title (original) language,Playlist create timestamp,Playlist update timestamp,Playlist video order,Playlist visibility
# PLc5oiabcabcabcabcabcrOvXgzabcabcm,False,,,,,display name of playlist,,2015-01-01T01:02:03+00:00,2022-10-28T02:23:15+00:00,Manual,Public' ---
# ##/documentation
# Create the playlist from the head content (if it seems to be valid):
csv_head = CSV.new(raw_head.strip('\n'), headers: true)
csv_head.next
if csv_head[11]
LOGGER.info("parse_playlist_export_csv: 03.1 raw_head is filled, doing dual csv playlist info scan. google takeout format after october 2023 (format seems to be one line of playlist metadata taken out of the google export's playlists.csv file, added above the actual playlist.csv content, separated by an empty line)")
title = csv_head[6]
description = "Playlist was imported from file '#{filename}'\n\nCreated on #{csv_head[8]}\nLast updated on #{csv_head[9]}\n" # TODO i8n
visibility = csv_head[11]
elsif csv_head[6]
LOGGER.info("parse_playlist_export_csv: 03.2 raw_head is filled, doing dual csv playlist info scan. google takeout format before october 2023 (roughly).")
title = csv_head[4]
description = csv_head[5]
visibility = csv_head[6]
else # we are using the defaults defined above instead.
LOGGER.info("parse_playlist_export_csv: 03.3 raw_head is filled, but in an unknown format. Using base defaults instead.")
end
if visibility.compare("Public", case_insensitive: true) == 0
privacy = PlaylistPrivacy::Public
else
privacy = PlaylistPrivacy::Private
end
else # choose a title from the provided upload filename
LOGGER.info("parse_playlist_export_csv: 03.2 raw_head is empty. Trying to guess fine-looking title and description from the provided upload filename.")
if (filename != "")
title = filename
end
nameendpos = filename.rindex(" videos.", filename.size) # XXX everything up to " videos.csv"
if !nameendpos # XXX everything up to file extension
nameendpos = filename.rindex(".", filename.size)
end
if nameendpos
title = filename[0, nameendpos]
end
end
playlist = create_playlist(title, privacy, user)
@ -56,7 +110,7 @@ struct Invidious::User
video_id = row[0]
if playlist
next if !video_id
next if video_id == "Video Id"
next if video_id == "Video Id" # TODO i8n
begin
video = get_video(video_id)
@ -115,7 +169,7 @@ struct Invidious::User
playlists.each do |item|
title = item["title"]?.try &.as_s?.try &.delete("<>")
description = item["description"]?.try &.as_s?.try &.delete("\r")
privacy = item["privacy"]?.try &.as_s?.try { |privacy| PlaylistPrivacy.parse? privacy }
privacy = item["privacy"]?.try &.as_s?.try { |raw_pl_privacy_state| PlaylistPrivacy.parse? raw_pl_privacy_state }
next if !title
next if !description
@ -161,7 +215,7 @@ struct Invidious::User
# Youtube
# -------------------
private def is_opml?(mimetype : String, extension : String)
private def opml?(mimetype : String, extension : String)
opml_mimetypes = [
"application/xml",
"text/xml",
@ -179,7 +233,7 @@ struct Invidious::User
def from_youtube(user : User, body : String, filename : String, type : String) : Bool
extension = filename.split(".").last
if is_opml?(type, extension)
if opml?(type, extension)
subscriptions = XML.parse(body)
user.subscriptions += subscriptions.xpath_nodes(%q(//outline[@type="rss"])).map do |channel|
channel["xmlUrl"].match!(/UC[a-zA-Z0-9_-]{22}/)[0]
@ -207,7 +261,7 @@ struct Invidious::User
extension = filename.split(".").last
if extension == "csv" || type == "text/csv"
playlist = parse_playlist_export_csv(user, body)
playlist = parse_playlist_export_csv(user, body, filename)
if playlist
return true
else

View File

@ -177,65 +177,8 @@ struct Video
# Misc. methods
def storyboards
storyboards = info.dig?("storyboards", "playerStoryboardSpecRenderer", "spec")
.try &.as_s.split("|")
if !storyboards
if storyboard = info.dig?("storyboards", "playerLiveStoryboardSpecRenderer", "spec").try &.as_s
return [{
url: storyboard.split("#")[0],
width: 106,
height: 60,
count: -1,
interval: 5000,
storyboard_width: 3,
storyboard_height: 3,
storyboard_count: -1,
}]
end
end
items = [] of NamedTuple(
url: String,
width: Int32,
height: Int32,
count: Int32,
interval: Int32,
storyboard_width: Int32,
storyboard_height: Int32,
storyboard_count: Int32)
return items if !storyboards
url = URI.parse(storyboards.shift)
params = HTTP::Params.parse(url.query || "")
storyboards.each_with_index do |sb, i|
width, height, count, storyboard_width, storyboard_height, interval, _, sigh = sb.split("#")
params["sigh"] = sigh
url.query = params.to_s
width = width.to_i
height = height.to_i
count = count.to_i
interval = interval.to_i
storyboard_width = storyboard_width.to_i
storyboard_height = storyboard_height.to_i
storyboard_count = (count / (storyboard_width * storyboard_height)).ceil.to_i
items << {
url: url.to_s.sub("$L", i).sub("$N", "M$M"),
width: width,
height: height,
count: count,
interval: interval,
storyboard_width: storyboard_width,
storyboard_height: storyboard_height,
storyboard_count: storyboard_count,
}
end
items
container = info.dig?("storyboards") || JSON::Any.new("{}")
return IV::Videos::Storyboard.from_yt_json(container, self.length_seconds)
end
def paid
@ -280,7 +223,7 @@ struct Video
info["genreUcid"].try &.as_s? ? "/channel/#{info["genreUcid"]}" : nil
end
def is_vr : Bool?
def vr? : Bool?
return {"EQUIRECTANGULAR", "MESH"}.includes? self.projection_type
end
@ -361,6 +304,21 @@ struct Video
{% if flag?(:debug_macros) %} {{debug}} {% end %}
end
# Macro to generate ? and = accessor methods for attributes in `info`
private macro predicate_bool(method_name, name)
# Return {{name.stringify}} from `info`
def {{method_name.id.underscore}}? : Bool
return info[{{name.stringify}}]?.try &.as_bool || false
end
# Update {{name.stringify}} into `info`
def {{method_name.id.underscore}}=(value : Bool)
info[{{name.stringify}}] = JSON::Any.new(value)
end
{% if flag?(:debug_macros) %} {{debug}} {% end %}
end
# Method definitions, using the macros above
getset_string author
@ -382,11 +340,12 @@ struct Video
getset_i64 likes
getset_i64 views
# TODO: Make predicate_bool the default as to adhere to Crystal conventions
getset_bool allowRatings
getset_bool authorVerified
getset_bool isFamilyFriendly
getset_bool isListed
getset_bool isUpcoming
predicate_bool upcoming, isUpcoming
end
def get_video(id, refresh = true, region = nil, force_refresh = false)

View File

@ -36,7 +36,13 @@ def parse_description(desc, video_id : String) : String?
return "" if content.empty?
commands = desc["commandRuns"]?.try &.as_a
return content if commands.nil?
if commands.nil?
# Slightly faster than HTML.escape, as we're only doing one pass on
# the string instead of five for the standard library
return String.build do |str|
copy_string(str, content.each_codepoint, content.size)
end
end
# Not everything is stored in UTF-8 on youtube's side. The SMP codepoints
# (0x10000 and above) are encoded as UTF-16 surrogate pairs, which are

View File

@ -0,0 +1,122 @@
require "uri"
require "http/params"
module Invidious::Videos
struct Storyboard
# Template URL
getter url : URI
getter proxied_url : URI
# Thumbnail parameters
getter width : Int32
getter height : Int32
getter count : Int32
getter interval : Int32
# Image (storyboard) parameters
getter rows : Int32
getter columns : Int32
getter images_count : Int32
def initialize(
*, @url, @width, @height, @count, @interval,
@rows, @columns, @images_count
)
authority = /(i\d?).ytimg.com/.match!(@url.host.not_nil!)[1]?
@proxied_url = URI.parse(HOST_URL)
@proxied_url.path = "/sb/#{authority}/#{@url.path.lchop("/sb/")}"
@proxied_url.query = @url.query
end
# Parse the JSON structure from Youtube
def self.from_yt_json(container : JSON::Any, length_seconds : Int32) : Array(Storyboard)
# Livestream storyboards are a bit different
# TODO: document exactly how
if storyboard = container.dig?("playerLiveStoryboardSpecRenderer", "spec").try &.as_s
return [Storyboard.new(
url: URI.parse(storyboard.split("#")[0]),
width: 106,
height: 60,
count: -1,
interval: 5000,
rows: 3,
columns: 3,
images_count: -1
)]
end
# Split the storyboard string into chunks
#
# General format (whitespaces added for legibility):
# https://i.ytimg.com/sb/<video_id>/storyboard3_L$L/$N.jpg?sqp=<sig0>
# | 48 # 27 # 100 # 10 # 10 # 0 # default # rs$<sig1>
# | 80 # 45 # 95 # 10 # 10 # 10000 # M$M # rs$<sig2>
# | 160 # 90 # 95 # 5 # 5 # 10000 # M$M # rs$<sig3>
#
storyboards = container.dig?("playerStoryboardSpecRenderer", "spec")
.try &.as_s.split("|")
return [] of Storyboard if !storyboards
# The base URL is the first chunk
base_url = URI.parse(storyboards.shift)
return storyboards.map_with_index do |sb, i|
# Separate the different storyboard parameters:
# width/height: respective dimensions, in pixels, of a single thumbnail
# count: how many thumbnails are displayed across the full video
# columns/rows: maximum amount of thumbnails that can be stuffed in a
# single image, horizontally and vertically.
# interval: interval between two thumbnails, in milliseconds
# name: storyboard filename. Usually "M$M" or "default"
# sigh: URL cryptographic signature
width, height, count, columns, rows, interval, name, sigh = sb.split("#")
width = width.to_i
height = height.to_i
count = count.to_i
interval = interval.to_i
columns = columns.to_i
rows = rows.to_i
# Copy base URL object, so that we can modify it
url = base_url.dup
# Add the signature to the URL
params = url.query_params
params["sigh"] = sigh
url.query_params = params
# Replace the template parts with what we have
url.path = url.path.sub("$L", i).sub("$N", name)
# This value represents the maximum amount of thumbnails that can fit
# in a single image. The last image (or the only one for short videos)
# will contain less thumbnails than that.
thumbnails_per_image = columns * rows
# This value represents the total amount of storyboards required to
# hold all of the thumbnails. It can't be less than 1.
images_count = (count / thumbnails_per_image).ceil.to_i
# Compute the interval when needed (in general, that's only required
# for the first "default" storyboard).
if interval == 0
interval = ((length_seconds / count) * 1_000).to_i
end
Storyboard.new(
url: url,
width: width,
height: height,
count: count,
interval: interval,
rows: rows,
columns: columns,
images_count: images_count,
)
end
end
end
end

View File

@ -110,13 +110,13 @@ module Invidious::Videos
"Language" => @language_code,
}
vtt = WebVTT.build(settings_field) do |vtt|
vtt = WebVTT.build(settings_field) do |builder|
@lines.each do |line|
# Section headers are excluded from the VTT conversion as to
# match the regular captions returned from YouTube as much as possible
next if line.is_a? HeadingLine
vtt.cue(line.start_ms, line.end_ms, line.line)
builder.cue(line.start_ms, line.end_ms, line.line)
end
end

View File

@ -6,4 +6,7 @@
title="<%= translate(locale, "search") %>"
value="<%= env.get?("search").try {|x| HTML.escape(x.as(String)) } %>">
</fieldset>
<button type="submit" id="searchbutton" aria-label="<%= translate(locale, "search") %>">
<i class="icon ion-ios-search"></i>
</button>
</form>

View File

@ -62,7 +62,7 @@ we're going to need to do it here in order to allow for translations.
"params" => params,
"preferences" => preferences,
"premiere_timestamp" => video.premiere_timestamp.try &.to_unix,
"vr" => video.is_vr,
"vr" => video.vr?,
"projection_type" => video.projection_type,
"local_disabled" => CONFIG.disabled?("local"),
"support_reddit" => true

View File

@ -1,6 +1,6 @@
def add_yt_headers(request)
request.headers.delete("User-Agent") if request.headers["User-Agent"] == "Crystal"
request.headers["User-Agent"] ||= "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
request.headers["User-Agent"] ||= "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
request.headers["Accept-Charset"] ||= "ISO-8859-1,utf-8;q=0.7,*;q=0.7"
request.headers["Accept"] ||= "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"

View File

@ -0,0 +1,121 @@
require "uri"
module UrlSanitizer
extend self
ALLOWED_QUERY_PARAMS = {
channel: ["u", "user", "lb"],
playlist: ["list"],
search: ["q", "search_query", "sp"],
watch: [
"v", # Video ID
"list", "index", # Playlist-related
"playlist", # Unnamed playlist (id,id,id,...) (embed-only?)
"t", "time_continue", "start", "end", # Timestamp
"lc", # Highlighted comment (watch page only)
],
}
# Returns whether the given string is an ASCII word. This is the same as
# running the following regex in US-ASCII locale: /^[\w-]+$/
private def ascii_word?(str : String) : Bool
return false if str.bytesize != str.size
str.each_byte do |byte|
next if 'a'.ord <= byte <= 'z'.ord
next if 'A'.ord <= byte <= 'Z'.ord
next if '0'.ord <= byte <= '9'.ord
next if byte == '-'.ord || byte == '_'.ord
return false
end
return true
end
# Return which kind of parameters are allowed based on the
# first path component (breadcrumb 0).
private def determine_allowed(path_root : String)
case path_root
when "watch", "w", "v", "embed", "e", "shorts", "clip"
return :watch
when .starts_with?("@"), "c", "channel", "user", "profile", "attribution_link"
return :channel
when "playlist", "mix"
return :playlist
when "results", "search"
return :search
else # hashtag, post, trending, brand URLs, etc..
return nil
end
end
# Create a new URI::Param containing only the allowed parameters
private def copy_params(unsafe_params : URI::Params, allowed_type) : URI::Params
new_params = URI::Params.new
ALLOWED_QUERY_PARAMS[allowed_type].each do |name|
if unsafe_params[name]?
# Only copy the last parameter, in case there is more than one
new_params[name] = unsafe_params.fetch_all(name)[-1]
end
end
return new_params
end
# Transform any user-supplied youtube URL into something we can trust
# and use across the code.
def process(str : String) : URI
# Because URI follows RFC3986 specifications, URL without a scheme
# will be parsed as a relative path. So we have to add a scheme ourselves.
str = "https://#{str}" if !str.starts_with?(/https?:\/\//)
unsafe_uri = URI.parse(str)
unsafe_host = unsafe_uri.host
unsafe_path = unsafe_uri.path
new_uri = URI.new(path: "/")
# Redirect to homepage for bogus URLs
return new_uri if (unsafe_host.nil? || unsafe_path.nil?)
breadcrumbs = unsafe_path
.split('/', remove_empty: true)
.compact_map do |bc|
# Exclude attempts at path trasversal
next if bc == "." || bc == ".."
# Non-alnum characters are unlikely in a genuine URL
next if !ascii_word?(bc)
bc
end
# If nothing remains, it's either a legit URL to the homepage
# (who does that!?) or because we filtered some junk earlier.
return new_uri if breadcrumbs.empty?
# Replace the original query parameters with the sanitized ones
case unsafe_host
when .ends_with?("youtube.com")
# Use our sanitized path (not forgetting the leading '/')
new_uri.path = "/#{breadcrumbs.join('/')}"
# Then determine which params are allowed, and copy them over
if allowed = determine_allowed(breadcrumbs[0])
new_uri.query_params = copy_params(unsafe_uri.query_params, allowed)
end
when "youtu.be"
# Always redirect to the watch page
new_uri.path = "/watch"
new_params = copy_params(unsafe_uri.query_params, :watch)
new_params["id"] = breadcrumbs[0]
new_uri.query_params = new_params
end
return new_uri
end
end

View File

@ -6,10 +6,10 @@ module YoutubeAPI
extend self
# For Android versions, see https://en.wikipedia.org/wiki/Android_version_history
private ANDROID_APP_VERSION = "19.14.42"
private ANDROID_USER_AGENT = "com.google.android.youtube/19.14.42 (Linux; U; Android 12; US) gzip"
private ANDROID_SDK_VERSION = 31_i64
private ANDROID_APP_VERSION = "19.32.34"
private ANDROID_VERSION = "12"
private ANDROID_USER_AGENT = "com.google.android.youtube/#{ANDROID_APP_VERSION} (Linux; U; Android #{ANDROID_VERSION}; US) gzip"
private ANDROID_SDK_VERSION = 31_i64
private ANDROID_TS_APP_VERSION = "1.9"
private ANDROID_TS_USER_AGENT = "com.google.android.youtube/1.9 (Linux; U; Android 12; US) gzip"
@ -17,9 +17,9 @@ module YoutubeAPI
# For Apple device names, see https://gist.github.com/adamawolf/3048717
# For iOS versions, see https://en.wikipedia.org/wiki/IOS_version_history#Releases,
# then go to the dedicated article of the major version you want.
private IOS_APP_VERSION = "19.16.3"
private IOS_USER_AGENT = "com.google.ios.youtube/19.16.3 (iPhone14,5; U; CPU iOS 17_4 like Mac OS X;)"
private IOS_VERSION = "17.4.0.21E219" # Major.Minor.Patch.Build
private IOS_APP_VERSION = "19.32.8"
private IOS_USER_AGENT = "com.google.ios.youtube/#{IOS_APP_VERSION} (iPhone14,5; U; CPU iOS 17_6 like Mac OS X;)"
private IOS_VERSION = "17.6.1.21G93" # Major.Minor.Patch.Build
private WINDOWS_VERSION = "10.0"
@ -48,7 +48,7 @@ module YoutubeAPI
ClientType::Web => {
name: "WEB",
name_proto: "1",
version: "2.20240304.00.00",
version: "2.20240814.00.00",
screen: "WATCH_FULL_SCREEN",
os_name: "Windows",
os_version: WINDOWS_VERSION,
@ -57,7 +57,7 @@ module YoutubeAPI
ClientType::WebEmbeddedPlayer => {
name: "WEB_EMBEDDED_PLAYER",
name_proto: "56",
version: "1.20240303.00.00",
version: "1.20240812.01.00",
screen: "EMBED",
os_name: "Windows",
os_version: WINDOWS_VERSION,
@ -66,7 +66,7 @@ module YoutubeAPI
ClientType::WebMobile => {
name: "MWEB",
name_proto: "2",
version: "2.20240304.08.00",
version: "2.20240813.02.00",
os_name: "Android",
os_version: ANDROID_VERSION,
platform: "MOBILE",
@ -74,7 +74,7 @@ module YoutubeAPI
ClientType::WebScreenEmbed => {
name: "WEB",
name_proto: "1",
version: "2.20240304.00.00",
version: "2.20240814.00.00",
screen: "EMBED",
os_name: "Windows",
os_version: WINDOWS_VERSION,
@ -147,8 +147,8 @@ module YoutubeAPI
ClientType::IOSMusic => {
name: "IOS_MUSIC",
name_proto: "26",
version: "6.42",
user_agent: "com.google.ios.youtubemusic/6.42 (iPhone14,5; U; CPU iOS 17_4 like Mac OS X;)",
version: "7.14",
user_agent: "com.google.ios.youtubemusic/7.14 (iPhone14,5; U; CPU iOS 17_6 like Mac OS X;)",
device_make: "Apple",
device_model: "iPhone14,5",
os_name: "iPhone",
@ -161,7 +161,7 @@ module YoutubeAPI
ClientType::TvHtml5 => {
name: "TVHTML5",
name_proto: "7",
version: "7.20240304.10.00",
version: "7.20240813.07.00",
},
ClientType::TvHtml5ScreenEmbed => {
name: "TVHTML5_SIMPLY_EMBEDDED_PLAYER",