Compare commits

..

74 Commits

Author SHA1 Message Date
Samantaz Fox
81fdf7b089
WIP 2024-07-25 22:26:08 +02:00
Samantaz Fox
d8dee8e767
Use new cache system for 'Video' objects 2024-07-25 22:26:08 +02:00
Samantaz Fox
c847e357f9
temp fix, while cacheable item is not integrated 2024-07-25 22:26:08 +02:00
Samantaz Fox
3a4ca20309
Misc: Add an utility function to for 'region' URL parameter 2024-07-25 22:26:08 +02:00
Samantaz Fox
8caa317c63
Cache: Create the base of the caching subsystem 2024-07-25 22:26:08 +02:00
Samantaz Fox
2d5575e036
Config: clean up the various converters 2024-07-25 22:26:08 +02:00
Samantaz Fox
a842f4358c
Config: Add scheme support to DBConfig 2024-07-25 22:26:08 +02:00
Samantaz Fox
eab6efc42a
Shards: Add required dependencies and update lock file 2024-07-25 22:26:08 +02:00
Samantaz Fox
325561e755
Channel: parse subscriber count and channel banner (#4785)
This PR adds support for parsing the newer channel header format
(banner + subscription parsing)

Before this change:
* 0 subscribers
* No banner image

After this change:
* Example with Mr Breast channel: 299M
* Image banner is visible

Closes issue 4783
2024-07-21 17:24:09 +02:00
Samantaz Fox
09bf09befe
Player: Fix playback position of already watched videos (#4731)
Trying to watch an already watched video will make the video start 15 seconds
before the end. This is not very comfortable when listening to music or
watching/listening playlists over and over.

This can be easily tested on any instance with the "Save playback position"
enabled in the Preferences.

Closes issue 3976
2024-07-21 17:24:06 +02:00
Samantaz Fox
7fdbda612f
Videos: Fix genre url being unusable (#4717)
Closes issue 4700
2024-07-21 17:24:03 +02:00
Samantaz Fox
4f60feee17
API: Fix out of bound error on empty playlists (#4696)
Before this PR, Invidious assumed that every playlist had at least one video.
When a playlist had no videos, Invidious was throwing an "Index out of bounds"
exception.

The following API endpoints were impacted:
* api/v1/playlists/:plid
* api/v1/auth/playlists/:plid

Fixes issue 4679
2024-07-21 17:24:01 +02:00
Samantaz Fox
733bd27a5c
Handle playlists cataloged as Podcast (#4695)
Videos of a playlist cataloged as podcast are called "episodes" therefore
Invidious was not able to find video in the text value inside the stats array.

Test case: "/playlist?list=PLDu-Eh5lUs1a4irCbnxMIB6FrUMaTXgVF"

Fixes issue 4688
2024-07-21 17:23:58 +02:00
Samantaz Fox
1ff0775f4b
API: Fix duplicated query parameters in proxied video URLs (#4587)
This pull request fixes that bug that was causing the query parameters to get
doubled in the streaming URLs when '?local=true' is passed to the
'/api/v1/videos/{id}' API endpoint.

Before: host/path?parameters?parameters
After: host/path?parameters

No associated open issue
2024-07-21 17:23:53 +02:00
Samantaz Fox
e62d4db752
API: Return actual stream height, width and fps (#4586)
At the moment Invidious will return hardcoded data for the 'size',
'qualityLabel' and 'fps' fields for streams, when such hardcoded data is
available, otherwise it just omits those fields from the response (e.g. with
the AV1 formats). Those issues are especially noticable when Invidious claims
that 50fps streams have 60fps and when it claims that the dimensions for a
vertical video are landscape. The DASH manifests that Invidious generates
already use the correct information.

This pull request corrects that issue by returning the information that
YouTube provides instead of hardcoded values and also fixes the long
standing bug of Invidious claiming that audio streams have 30 fps.

Here are two test cases:
50/25/13fps: https://youtu.be/GbXYZwUigCM (/api/v1/videos/GbXYZwUigCM)
vertical video: https://youtu.be/hxQwWEOOyU8 (/api/v1/videos/hxQwWEOOyU8)

Originally these problems were going to be solved by the complete refactor
of stream handling in 3620, but as that pull request got closed by the stale
bot over a month ago and has such a massive scope that it would require a
massive amount of work to complete it, I decided to open this pull request
that takes a less radical approach of just fixing bugs instead of a full
on refactoring.

FreeTube generates it's own DASH manifests instead of using Invidious' one,
so that it can support multiple audio tracks and HDR. Unfortunately due to
the missing and inaccurate information in the API responses, FreeTube has
to request the DASH manifest from Invidious to extract the height, width and
fps. With this pull request FreeTube could rely just on the API response,
saving that extra request to the Invidious instance. It would also make it
possible for FreeTube to use the vp9 streams with Invidious, which would
reduce the load on the video proxies.

Closes issue 4131
2024-07-21 17:23:50 +02:00
Samantaz Fox
8b1da2001e
Preferences: Fix handling of modified source code URL(#4437)
Before this PR, setting the modified code repo URL through the preferences
page in Invidious was broken:

* the HTML input tag for this field had invalid type "input"
  (though browser falls back on text input)

* the URL was used to set the "checked" property and not as a plain value,
  which makes no sense for a text-based input (and resulted in a blank field)

* when the submitted field is empty, the retrieved value was an empty 'String'
  instead of 'nil', causing the "modified source code URL" to be an empty
  'href' link which just pointed to the current page

No associated open issue
2024-07-21 17:23:48 +02:00
Samantaz Fox
5a12005b48
API: Fix URL for vtt subtitles (#4221)
For 'fmt=vtt' to work, the 'fmt' parameter needs to be replaced
in the original caption api URL.

No associated open issue
2024-07-21 17:23:44 +02:00
Samantaz Fox
bad92093bf
Channels: Add sort options to streams (#4224) 2024-07-10 22:28:22 +02:00
Samantaz Fox
436a61e3bb
API: Fix error code for disabled popular endpoint (#4296)
When visiting /api/v1/popular and popular endpoint is disabled
Before:

500 {"error":"Closed stream"}

After

403 {"error":"Administrator has disabled this endpoint."}
2024-07-10 22:25:31 +02:00
Samantaz Fox
5e0f55333a
Allow embedding videos in local HTML files (#4450)
The current Content Security Policy does not allow to embed videos
inside local HTML files which are viewed in the browser via the file
protocol. This commit adds the file protocol to the allowed frame
ancestors, so that the embedded videos load correctly in local HTML
files.

This behaviour is consistent which how the official YouTube website
allows to embed videos from itself.

Closes issue 4448
2024-07-10 22:24:18 +02:00
Samantaz Fox
de61b163a3
CI: Bump Crystal version matrix (#4654) 2024-07-10 22:21:17 +02:00
Samantaz Fox
99c7e9e800
YtAPI: Remove API keys like official clients (#4655)
This PR removes API keys from innertube requests, as the official clients
did it too.
2024-07-10 22:19:51 +02:00
Samantaz Fox
e9bab06e90
HTML: Use full URL in the og:image property (#4675)
Some opengraph implementations don't support a URL without the domain
therefore failing to fetch the video thumbnail and channel image.
This pull request basically fixes that.
2024-07-10 22:17:45 +02:00
Samantaz Fox
a56a724a55
Rewrite transcript logic to be more generic (#4747)
The transcript logic in Invidious was written specifically as a workaround for
captions, and not transcripts as a feature.

This PR genericises the logic as so it can be used to implement transcripts
within Invidious.

The most notable change is the added parsing of section headings when it was
previously skipped over in favor of regular lines.
2024-07-10 22:14:56 +02:00
Samantaz Fox
0a54e26536
CI: Run Ameba (#4753)
This PR simply adds Ameba to the CI but doesn't actually fix any of the
detected issues.
2024-07-10 22:13:45 +02:00
Samantaz Fox
d135e5b7f7
CI: Add release based containers (#4763)
This PR changes the current master based container to use "master" tag instead
of "latest" tag and adds a new workflow to build a container on each new
release which has the "latest" tag, and a tag based on the current released
version.
2024-07-10 22:11:01 +02:00
ChunkyProgrammer
911dad6935 Channel: parse subscriber count and channel banner 2024-07-09 14:43:14 -04:00
syeopite
220cc9bd2f
Typo
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
2024-07-04 10:14:19 -07:00
syeopite
aace30b2b4
Bump nightly container build workflow crystal ver 2024-07-04 10:11:36 -07:00
syeopite
64d1f26ece
Fix trigger for stable container build 2024-07-01 21:39:14 -07:00
syeopite
8f5c6a602b
Rename container workflows 2024-07-01 21:35:08 -07:00
syeopite
dd38eef41a
Add workflow to build container on release 2024-06-24 11:45:00 -07:00
syeopite
848ab1e9c8
Specify which workflow builds from master 2024-06-24 11:36:11 -07:00
syeopite
933802b897
Use "master" label for master container build 2024-06-24 11:34:55 -07:00
meatball
3bac467a8c Call as? instead of as to not force string conversion 2024-06-19 12:52:53 +02:00
meatball
248df785d7 Update spec and rollback to last commits changes 2024-06-18 20:55:14 +02:00
syeopite
6b429575bf
Update ameba version 2024-06-16 16:22:01 -07:00
syeopite
e0ed094cc4
Cache ameba binary 2024-06-16 13:29:06 -07:00
syeopite
a644d76497
Update ameba config 2024-06-16 13:21:55 -07:00
syeopite
45fd4a1968
Add job to lint code through Ameba in CI 2024-06-16 13:21:55 -07:00
Fijxu
e82c965e89
Player: Fix video playback for videos that have already been watched.
Trying to watch an already watched video will make the video start 15
seconds before the end of the video. This is not very comfortable when
listening to music or watching/listening playlists over and over.
2024-06-15 18:15:51 -04:00
syeopite
f466116cd7
Extract label for transcript in YouTube response 2024-06-13 09:07:20 -07:00
syeopite
5b519123a7
Raise error when transcript does not exist 2024-06-11 18:46:34 -07:00
syeopite
0224162ad2
Rewrite transcript logic to be more generic
The transcript logic in Invidious was written specifically
as a workaround for captions, and not transcripts as a feature.

This commit genericises the logic a bit as so it can be used for
implementing transcripts within Invidious' API and UI as well.

The most notable change is the added parsing of section headings
when it was previously skipped over in favor of regular lines.
2024-06-11 18:23:01 -07:00
meatball
04ca64691b Make solution complaint with spec 2024-05-30 22:37:55 +02:00
meatball
5957523624 Improve code quallity 2024-05-30 22:13:30 +02:00
meatball
629599f940 Fix change in parser file 2024-05-30 21:57:15 +02:00
meatball
31ad708206 fix: Handle nil value for genreUcid in Video struct 2024-05-30 21:56:33 +02:00
absidue
3b773c4f77 Fix missing commas 2024-05-14 19:02:41 +02:00
absidue
57e606cb43 Add back missing resolution field 2024-05-14 19:02:41 +02:00
absidue
f57aac5815 Fix the missing p in the quality labels.
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
2024-05-14 19:02:41 +02:00
absidue
71a821a7e6 Return actual height, width and fps for streams in /api/v1/videos 2024-05-14 19:02:32 +02:00
Fijxu
e0d0dbde3c
API: Check if playlist has any videos on it.
Invidious assumes that every playlist will have at least one video
because it needs to check for the `index` key. So if there is no videos
on a playlist, there is no `index` key and Invidious throws
`Index out of bounds`
2024-05-13 21:07:46 -04:00
Fijxu
90fcf80a8d
Handle playlists cataloged as Podcast
Videos of a playlist cataloged as podcast are called episodes therefore
Invidious was not able to find `video` in the `text` value inside the
stats array.
2024-05-13 19:39:46 -04:00
Fijxu
9d66676f2d
Use full URL in the og:image property. 2024-05-01 22:21:18 -04:00
Samantaz Fox
2fdb6dd644
CI: Bump Crystal version in docker too 2024-04-27 21:02:37 +02:00
Samantaz Fox
470245de54
YtAPI: Remove API keys like official clients 2024-04-27 20:48:42 +02:00
Samantaz Fox
b0ec359028
CI: Bump Crystal version matrix 2024-04-27 20:01:19 +02:00
absidue
b90cf286fc Fix duplicate query parameters in URLs when local=true for /api/v1/videos/{id} 2024-04-20 20:46:01 +02:00
Brahim Hadriche
a9e8aabe1f Merge commit '08390acd0c17875fddb84cabba54197a5b5740e4' into fix/popular-disabled-error 2024-04-01 10:03:37 -04:00
Brahim Hadriche
b0c6bdf44c use 403 code 2024-04-01 10:03:29 -04:00
Brahim Hadriche
c5eb10b21f Revert "Fix error code for disabled popular endpoint"
This reverts commit 1363fb8094.
2024-04-01 10:02:49 -04:00
src-tinkerer
72fe8af850
Merge branch 'master' into stream-sort 2024-03-26 12:19:45 +00:00
nooptek
499aed37dd Fix handling of modified source code URL setting 2024-03-10 17:51:29 +01:00
Tomasz Wilczyński
4adb4c00d2
routes: Allow embedding videos in local HTML files (fixes #4448)
The current Content Security Policy does not allow to embed videos
inside local HTML files which are viewed in the browser via the file
protocol. This commit adds the file protocol to the allowed frame
ancestors, so that the embedded videos load correctly in local HTML
files.

This behaviour is consistent which how the official YouTube website
allows to embed videos from itself.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-02-24 20:01:16 +01:00
src-tinkerer
cf61af67ab Update src/invidious/routes/channels.cr sort_by for consistency 2023-11-30 14:34:01 +03:30
Brahim Hadriche
1363fb8094 Fix error code for disabled popular endpoint 2023-11-28 21:34:17 -05:00
src-tinkerer
5f2b43d653 Remove unecessary if condition in videos.cr 2023-11-25 00:48:27 +03:30
src-tinkerer
6251d8d43f Rename a variable in videos.cr 2023-11-25 00:46:11 +03:30
src-tinkerer
162b89d942 Fix format in videos.cr 2023-11-23 14:44:37 +03:30
src-tinkerer
0d63ad5a7f Use a single function for fetching channel contents 2023-11-22 14:52:17 +03:30
src-tinkerer
63e5d72466 Remove unused function produce_channel_livestream_url 2023-11-20 15:50:59 +03:30
karelrooted
c251c66748 fix youtube api vtt format subtitle
for fmt=vtt to work the fmt parameter in the original caption api url need to be replaced
2023-11-14 13:16:08 +08:00
src-tinkerer
b0df3774db Add sort options to streams 2023-11-01 21:56:25 +03:30
30 changed files with 341 additions and 192 deletions

View File

@ -20,6 +20,9 @@ Lint/ShadowingOuterLocalVar:
Excluded:
- src/invidious/helpers/tokens.cr
Lint/NotNil:
Enabled: false
#
# Style
@ -31,6 +34,13 @@ Style/RedundantBegin:
Style/RedundantReturn:
Enabled: false
Style/ParenthesesAroundCondition:
Enabled: false
# This requires a rewrite of most data structs (and their usage) in Invidious.
Style/QueryBoolMethods:
Enabled: false
#
# Metrics
@ -39,50 +49,4 @@ Style/RedundantReturn:
# Ignore function complexity (number of if/else & case/when branches)
# For some functions that can hardly be simplified for now
Metrics/CyclomaticComplexity:
Excluded:
# get_about_info(ucid, locale) => [17/10]
- src/invidious/channels/about.cr
# fetch_channel_community(ucid, continuation, ...) => [34/10]
- src/invidious/channels/community.cr
# create_notification_stream(env, topics, connection_channel) => [14/10]
- src/invidious/helpers/helpers.cr:84:5
# get_index(plural_form, count) => [25/10]
- src/invidious/helpers/i18next.cr
# call(context) => [18/10]
- src/invidious/helpers/static_file_handler.cr
# show(env) => [38/10]
- src/invidious/routes/embed.cr
# get_video_playback(env) => [45/10]
- src/invidious/routes/video_playback.cr
# handle(env) => [40/10]
- src/invidious/routes/watch.cr
# playlist_ajax(env) => [24/10]
- src/invidious/routes/playlists.cr
# fetch_youtube_comments(id, cursor, ....) => [40/10]
# template_youtube_comments(comments, locale, ...) => [16/10]
# content_to_comment_html(content) => [14/10]
- src/invidious/comments.cr
# to_json(locale, json) => [21/10]
# extract_video_info(video_id, ...) => [44/10]
# process_video_params(query, preferences) => [20/10]
- src/invidious/videos.cr
#src/invidious/playlists.cr:327:5
#[C] Metrics/CyclomaticComplexity: Cyclomatic complexity too high [19/10]
# fetch_playlist(plid : String)
#src/invidious/playlists.cr:436:5
#[C] Metrics/CyclomaticComplexity: Cyclomatic complexity too high [11/10]
# extract_playlist_videos(initial_data : Hash(String, JSON::Any))
Enabled: false

View File

@ -1,4 +1,4 @@
name: Build and release container
name: Build and release container directly from master
on:
push:
@ -24,9 +24,9 @@ jobs:
uses: actions/checkout@v4
- name: Install Crystal
uses: crystal-lang/install-crystal@v1.8.0
uses: crystal-lang/install-crystal@v1.8.2
with:
crystal: 1.9.2
crystal: 1.12.2
- name: Run lint
run: |
@ -58,7 +58,7 @@ jobs:
images: quay.io/invidious/invidious
tags: |
type=sha,format=short,prefix={{date 'YYYY.MM.DD'}}-,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=master,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
labels: |
quay.expires-after=12w
@ -83,7 +83,7 @@ jobs:
suffix=-arm64
tags: |
type=sha,format=short,prefix={{date 'YYYY.MM.DD'}}-,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=master,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
labels: |
quay.expires-after=12w

View File

@ -0,0 +1,90 @@
name: Build and release container
on:
push:
tags:
- "v*"
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Crystal
uses: crystal-lang/install-crystal@v1.8.2
with:
crystal: 1.12.2
- name: Run lint
run: |
if ! crystal tool format --check; then
crystal tool format
git diff
exit 1
fi
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
with:
platforms: arm64
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to registry
uses: docker/login-action@v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: quay.io/invidious/invidious
tags: |
type=semver,pattern={{version}}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
labels: |
quay.expires-after=12w
- name: Build and push Docker AMD64 image for Push Event
uses: docker/build-push-action@v5
with:
context: .
file: docker/Dockerfile
platforms: linux/amd64
labels: ${{ steps.meta.outputs.labels }}
push: true
tags: ${{ steps.meta.outputs.tags }}
build-args: |
"release=1"
- name: Docker meta
id: meta-arm64
uses: docker/metadata-action@v5
with:
images: quay.io/invidious/invidious
flavor: |
suffix=-arm64
tags: |
type=semver,pattern={{version}}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
labels: |
quay.expires-after=12w
- name: Build and push Docker ARM64 image for Push Event
uses: docker/build-push-action@v5
with:
context: .
file: docker/Dockerfile.arm64
platforms: linux/arm64/v8
labels: ${{ steps.meta-arm64.outputs.labels }}
push: true
tags: ${{ steps.meta-arm64.outputs.tags }}
build-args: |
"release=1"

View File

@ -38,10 +38,10 @@ jobs:
matrix:
stable: [true]
crystal:
- 1.7.3
- 1.8.2
- 1.9.2
- 1.10.1
- 1.11.2
- 1.12.1
include:
- crystal: nightly
stable: false
@ -124,4 +124,28 @@ jobs:
- name: Test Docker
run: while curl -Isf http://localhost:3000; do sleep 1; done
ameba_lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Install Crystal
uses: crystal-lang/install-crystal@v1.8.0
with:
crystal: latest
- name: Cache Shards
uses: actions/cache@v3
with:
path: |
./lib
./bin
key: shards-${{ hashFiles('shard.lock') }}
- name: Install Shards
run: shards install
- name: Run Ameba linter
run: bin/ameba

View File

@ -351,7 +351,12 @@ if (video_data.params.save_player_pos) {
const rememberedTime = get_video_time();
let lastUpdated = 0;
if(!hasTimeParam) set_seconds_after_start(rememberedTime);
if(!hasTimeParam) {
if (rememberedTime >= video_data.length_seconds - 20)
set_seconds_after_start(0);
else
set_seconds_after_start(rememberedTime);
}
player.on('timeupdate', function () {
const raw = player.currentTime();

View File

@ -1,4 +1,4 @@
FROM crystallang/crystal:1.8.2-alpine AS builder
FROM crystallang/crystal:1.12.1-alpine AS builder
RUN apk add --no-cache sqlite-static yaml-static

View File

@ -1,5 +1,5 @@
FROM alpine:3.18 AS builder
RUN apk add --no-cache 'crystal=1.8.2-r0' shards sqlite-static yaml-static yaml-dev libxml2-static zlib-static openssl-libs-static openssl-dev musl-dev xz-static
FROM alpine:3.19 AS builder
RUN apk add --no-cache 'crystal=1.10.1-r0' shards sqlite-static yaml-static yaml-dev libxml2-static zlib-static openssl-libs-static openssl-dev musl-dev xz-static
ARG release

View File

@ -2,7 +2,7 @@ version: 2.0
shards:
ameba:
git: https://github.com/crystal-ameba/ameba.git
version: 1.5.0
version: 1.6.1
athena-negotiation:
git: https://github.com/athena-framework/negotiation.git

View File

@ -45,7 +45,7 @@ development_dependencies:
version: ~> 0.10.4
ameba:
github: crystal-ameba/ameba
version: ~> 1.5.0
version: ~> 1.6.1
crystal: ">= 1.0.0, < 2.0.0"

View File

@ -67,7 +67,7 @@ Spectator.describe "parse_video_info" do
# Video metadata
expect(info["genre"].as_s).to eq("Entertainment")
expect(info["genreUcid"].as_s).to be_empty
expect(info["genreUcid"].as_s?).to be_nil
expect(info["license"].as_s).to be_empty
# Author infos
@ -151,7 +151,7 @@ Spectator.describe "parse_video_info" do
# Video metadata
expect(info["genre"].as_s).to eq("Music")
expect(info["genreUcid"].as_s).to be_empty
expect(info["genreUcid"].as_s?).to be_nil
expect(info["license"].as_s).to be_empty
# Author infos

View File

@ -94,7 +94,7 @@ Spectator.describe "parse_video_info" do
# Video metadata
expect(info["genre"].as_s).to eq("Entertainment")
expect(info["genreUcid"].as_s).to be_empty
expect(info["genreUcid"].as_s?).to be_nil
expect(info["license"].as_s).to be_empty
# Author infos

View File

@ -72,6 +72,7 @@ def get_about_info(ucid, locale) : AboutChannel
# Raises a KeyError on failure.
banners = initdata["header"]["c4TabbedHeaderRenderer"]?.try &.["banner"]?.try &.["thumbnails"]?
banners ||= initdata.dig?("header", "pageHeaderRenderer", "content", "pageHeaderViewModel", "banner", "imageBannerViewModel", "image", "sources")
banner = banners.try &.[-1]?.try &.["url"].as_s?
# if banner.includes? "channels/c4/default_banner"
@ -147,9 +148,17 @@ def get_about_info(ucid, locale) : AboutChannel
end
end
sub_count = initdata
.dig?("header", "c4TabbedHeaderRenderer", "subscriberCountText", "simpleText").try &.as_s?
.try { |text| short_text_to_number(text.split(" ")[0]).to_i32 } || 0
sub_count = 0
if (metadata_rows = initdata.dig?("header", "pageHeaderRenderer", "content", "pageHeaderViewModel", "metadata", "contentMetadataViewModel", "metadataRows").try &.as_a)
metadata_rows.each do |row|
metadata_part = row.dig?("metadataParts").try &.as_a.find { |i| i.dig?("text", "content").try &.as_s.includes?("subscribers") }
if !metadata_part.nil?
sub_count = short_text_to_number(metadata_part.dig("text", "content").as_s.split(" ")[0]).to_i32
end
break if sub_count != 0
end
end
AboutChannel.new(
ucid: ucid,

View File

@ -1,4 +1,4 @@
def produce_channel_videos_continuation(ucid, page = 1, auto_generated = nil, sort_by = "newest", v2 = false)
def produce_channel_content_continuation(ucid, content_type, page = 1, auto_generated = nil, sort_by = "newest", v2 = false)
object_inner_2 = {
"2:0:embedded" => {
"1:0:varint" => 0_i64,
@ -16,6 +16,13 @@ def produce_channel_videos_continuation(ucid, page = 1, auto_generated = nil, so
.try { |i| Base64.urlsafe_encode(i) }
.try { |i| URI.encode_www_form(i) }
content_type_numerical =
case content_type
when "videos" then 15
when "livestreams" then 14
else 15 # Fallback to "videos"
end
sort_by_numerical =
case sort_by
when "newest" then 1_i64
@ -27,7 +34,7 @@ def produce_channel_videos_continuation(ucid, page = 1, auto_generated = nil, so
object_inner_1 = {
"110:embedded" => {
"3:embedded" => {
"15:embedded" => {
"#{content_type_numerical}:embedded" => {
"1:embedded" => {
"1:string" => object_inner_2_encoded,
},
@ -62,6 +69,10 @@ def produce_channel_videos_continuation(ucid, page = 1, auto_generated = nil, so
return continuation
end
def make_initial_content_ctoken(ucid, content_type, sort_by) : String
return produce_channel_content_continuation(ucid, content_type, sort_by: sort_by)
end
module Invidious::Channel::Tabs
extend self
@ -69,10 +80,6 @@ module Invidious::Channel::Tabs
# Regular videos
# -------------------
def make_initial_video_ctoken(ucid, sort_by) : String
return produce_channel_videos_continuation(ucid, sort_by: sort_by)
end
# Wrapper for AboutChannel, as we still need to call get_videos with
# an author name and ucid directly (e.g in RSS feeds).
# TODO: figure out how to get rid of that
@ -94,7 +101,7 @@ module Invidious::Channel::Tabs
end
def get_videos(author : String, ucid : String, *, continuation : String? = nil, sort_by = "newest")
continuation ||= make_initial_video_ctoken(ucid, sort_by)
continuation ||= make_initial_content_ctoken(ucid, "videos", sort_by)
initial_data = YoutubeAPI.browse(continuation: continuation)
return extract_items(initial_data, author, ucid)
@ -138,21 +145,18 @@ module Invidious::Channel::Tabs
# Livestreams
# -------------------
def get_livestreams(channel : AboutChannel, continuation : String? = nil)
if continuation.nil?
# EgdzdHJlYW1z8gYECgJ6AA%3D%3D is the protobuf object to load "streams"
initial_data = YoutubeAPI.browse(channel.ucid, params: "EgdzdHJlYW1z8gYECgJ6AA%3D%3D")
else
initial_data = YoutubeAPI.browse(continuation: continuation)
end
def get_livestreams(channel : AboutChannel, continuation : String? = nil, sort_by = "newest")
continuation ||= make_initial_content_ctoken(channel.ucid, "livestreams", sort_by)
initial_data = YoutubeAPI.browse(continuation: continuation)
return extract_items(initial_data, channel.author, channel.ucid)
end
def get_60_livestreams(channel : AboutChannel, continuation : String? = nil)
def get_60_livestreams(channel : AboutChannel, *, continuation : String? = nil, sort_by = "newest")
if continuation.nil?
# Fetch the first "page" of streams
items, next_continuation = get_livestreams(channel)
# Fetch the first "page" of stream
items, next_continuation = get_livestreams(channel, sort_by: sort_by)
else
# Fetch a "page" of streams using the given continuation token
items, next_continuation = get_livestreams(channel, continuation: continuation)

View File

@ -11,11 +11,12 @@ module Invidious::HttpServer
params = url.query_params
params["host"] = url.host.not_nil! # Should never be nil, in theory
params["region"] = region if !region.nil?
url.query_params = params
if absolute
return "#{HOST_URL}#{url.request_target}?#{params}"
return "#{HOST_URL}#{url.request_target}"
else
return "#{url.request_target}?#{params}"
return url.request_target
end
end

View File

@ -114,25 +114,31 @@ module Invidious::JSONify::APIv1
json.field "projectionType", fmt["projectionType"]
if fmt_info = Invidious::Videos::Formats.itag_to_metadata?(fmt["itag"])
fps = fmt_info["fps"]?.try &.to_i || fmt["fps"]?.try &.as_i || 30
height = fmt["height"]?.try &.as_i
width = fmt["width"]?.try &.as_i
fps = fmt["fps"]?.try &.as_i
if fps
json.field "fps", fps
end
if height && width
json.field "size", "#{width}x#{height}"
json.field "resolution", "#{height}p"
quality_label = "#{width > height ? height : width}p"
if fps && fps > 30
quality_label += fps.to_s
end
json.field "qualityLabel", quality_label
end
if fmt_info = Invidious::Videos::Formats.itag_to_metadata?(fmt["itag"])
json.field "container", fmt_info["ext"]
json.field "encoding", fmt_info["vcodec"]? || fmt_info["acodec"]
if fmt_info["height"]?
json.field "resolution", "#{fmt_info["height"]}p"
quality_label = "#{fmt_info["height"]}p"
if fps > 30
quality_label += "60"
end
json.field "qualityLabel", quality_label
if fmt_info["width"]?
json.field "size", "#{fmt_info["width"]}x#{fmt_info["height"]}"
end
end
end
# Livestream chunk infos
@ -163,26 +169,31 @@ module Invidious::JSONify::APIv1
json.field "bitrate", fmt["bitrate"].as_i.to_s if fmt["bitrate"]?
fmt_info = Invidious::Videos::Formats.itag_to_metadata?(fmt["itag"])
if fmt_info
fps = fmt_info["fps"]?.try &.to_i || fmt["fps"]?.try &.as_i || 30
height = fmt["height"]?.try &.as_i
width = fmt["width"]?.try &.as_i
fps = fmt["fps"]?.try &.as_i
if fps
json.field "fps", fps
end
if height && width
json.field "size", "#{width}x#{height}"
json.field "resolution", "#{height}p"
quality_label = "#{width > height ? height : width}p"
if fps && fps > 30
quality_label += fps.to_s
end
json.field "qualityLabel", quality_label
end
if fmt_info = Invidious::Videos::Formats.itag_to_metadata?(fmt["itag"])
json.field "container", fmt_info["ext"]
json.field "encoding", fmt_info["vcodec"]? || fmt_info["acodec"]
if fmt_info["height"]?
json.field "resolution", "#{fmt_info["height"]}p"
quality_label = "#{fmt_info["height"]}p"
if fps > 30
quality_label += "60"
end
json.field "qualityLabel", quality_label
if fmt_info["width"]?
json.field "size", "#{fmt_info["width"]}x#{fmt_info["height"]}"
end
end
end
end
end

View File

@ -366,6 +366,8 @@ def fetch_playlist(plid : String)
if text.includes? "video"
video_count = text.gsub(/\D/, "").to_i? || 0
elsif text.includes? "episode"
video_count = text.gsub(/\D/, "").to_i? || 0
elsif text.includes? "view"
views = text.gsub(/\D/, "").to_i64? || 0_i64
else

View File

@ -208,11 +208,12 @@ module Invidious::Routes::API::V1::Channels
get_channel()
# Retrieve continuation from URL parameters
sort_by = env.params.query["sort_by"]?.try &.downcase || "newest"
continuation = env.params.query["continuation"]?
begin
videos, next_continuation = Channel::Tabs.get_60_livestreams(
channel, continuation: continuation
channel, continuation: continuation, sort_by: sort_by
)
rescue ex
return error_json(500, ex)

View File

@ -31,7 +31,7 @@ module Invidious::Routes::API::V1::Feeds
if !CONFIG.popular_enabled
error_message = {"error" => "Administrator has disabled this endpoint."}.to_json
haltf env, 400, error_message
haltf env, 403, error_message
end
JSON.build do |json|

View File

@ -74,7 +74,9 @@ module Invidious::Routes::API::V1::Misc
response = playlist.to_json(offset, video_id: video_id)
json_response = JSON.parse(response)
if json_response["videos"].as_a[0]["index"] != offset
if json_response["videos"].as_a.empty?
json_response = JSON.parse(response)
elsif json_response["videos"].as_a[0]["index"] != offset
offset = json_response["videos"].as_a[0]["index"].as_i
lookback = offset < 50 ? offset : 50
response = playlist.to_json(offset - lookback)

View File

@ -89,9 +89,14 @@ module Invidious::Routes::API::V1::Videos
if CONFIG.use_innertube_for_captions
params = Invidious::Videos::Transcript.generate_param(id, caption.language_code, caption.auto_generated)
initial_data = YoutubeAPI.get_transcript(params)
webvtt = Invidious::Videos::Transcript.convert_transcripts_to_vtt(initial_data, caption.language_code)
transcript = Invidious::Videos::Transcript.from_raw(
YoutubeAPI.get_transcript(params),
caption.language_code,
caption.auto_generated
)
webvtt = transcript.to_vtt
else
# Timedtext API handling
url = URI.parse("#{caption.base_url}&tlang=#{tlang}").request_target
@ -136,7 +141,11 @@ module Invidious::Routes::API::V1::Videos
end
end
else
webvtt = YT_POOL.client &.get("#{url}&fmt=vtt").body
uri = URI.parse(url)
query_params = uri.query_params
query_params["fmt"] = "vtt"
uri.query_params = query_params
webvtt = YT_POOL.client &.get(uri.request_target).body
if webvtt.starts_with?("<?xml")
webvtt = caption.timedtext_to_vtt(webvtt)

View File

@ -30,7 +30,7 @@ module Invidious::Routes::BeforeAll
# Only allow the pages at /embed/* to be embedded
if env.request.resource.starts_with?("/embed")
frame_ancestors = "'self' http: https:"
frame_ancestors = "'self' file: http: https:"
else
frame_ancestors = "'none'"
end

View File

@ -81,13 +81,12 @@ module Invidious::Routes::Channels
return env.redirect "/channel/#{channel.ucid}"
end
# TODO: support sort option for livestreams
sort_by = ""
sort_options = [] of String
sort_by = env.params.query["sort_by"]?.try &.downcase || "newest"
sort_options = {"newest", "oldest", "popular"}
# Fetch items and continuation token
items, next_continuation = Channel::Tabs.get_60_livestreams(
channel, continuation: continuation
channel, continuation: continuation, sort_by: sort_by
)
selected_tab = Frontend::ChannelPage::TabsAvailable::Streams

View File

@ -214,7 +214,7 @@ module Invidious::Routes::PreferencesRoute
statistics_enabled ||= "off"
CONFIG.statistics_enabled = statistics_enabled == "on"
CONFIG.modified_source_code_url = env.params.body["modified_source_code_url"]?.try &.as(String)
CONFIG.modified_source_code_url = env.params.body["modified_source_code_url"]?.presence
File.write("config/config.yml", CONFIG.to_yaml)
end

View File

@ -280,7 +280,7 @@ struct Video
end
def genre_url : String?
info["genreUcid"]? ? "/channel/#{info["genreUcid"]}" : nil
info["genreUcid"].try &.as_s? ? "/channel/#{info["genreUcid"]}" : nil
end
def is_vr : Bool?

View File

@ -424,7 +424,7 @@ def parse_video_info(video_id : String, player_response : Hash(String, JSON::Any
"shortDescription" => JSON::Any.new(short_description.try &.as_s || nil),
# Video metadata
"genre" => JSON::Any.new(genre.try &.as_s || ""),
"genreUcid" => JSON::Any.new(genre_ucid.try &.as_s || ""),
"genreUcid" => JSON::Any.new(genre_ucid.try &.as_s?),
"license" => JSON::Any.new(license.try &.as_s || ""),
# Music section
"music" => JSON.parse(music_list.to_json),

View File

@ -1,8 +1,26 @@
module Invidious::Videos
# Namespace for methods primarily relating to Transcripts
module Transcript
record TranscriptLine, start_ms : Time::Span, end_ms : Time::Span, line : String
# A `Transcripts` struct encapsulates a sequence of lines that together forms the whole transcript for a given YouTube video.
# These lines can be categorized into two types: section headings and regular lines representing content from the video.
struct Transcript
# Types
record HeadingLine, start_ms : Time::Span, end_ms : Time::Span, line : String
record RegularLine, start_ms : Time::Span, end_ms : Time::Span, line : String
alias TranscriptLine = HeadingLine | RegularLine
property lines : Array(TranscriptLine)
property language_code : String
property auto_generated : Bool
# User friendly label for the current transcript.
# Example: "English (auto-generated)"
property label : String
# Initializes a new Transcript struct with the contents and associated metadata describing it
def initialize(@lines : Array(TranscriptLine), @language_code : String, @auto_generated : Bool, @label : String)
end
# Generates a protobuf string to fetch the requested transcript from YouTube
def self.generate_param(video_id : String, language_code : String, auto_generated : Bool) : String
kind = auto_generated ? "asr" : ""
@ -30,48 +48,79 @@ module Invidious::Videos
return params
end
def self.convert_transcripts_to_vtt(initial_data : Hash(String, JSON::Any), target_language : String) : String
# Convert into array of TranscriptLine
lines = self.parse(initial_data)
# Constructs a Transcripts struct from the initial YouTube response
def self.from_raw(initial_data : Hash(String, JSON::Any), language_code : String, auto_generated : Bool)
transcript_panel = initial_data.dig("actions", 0, "updateEngagementPanelAction", "content", "transcriptRenderer",
"content", "transcriptSearchPanelRenderer")
segment_list = transcript_panel.dig("body", "transcriptSegmentListRenderer")
if !segment_list["initialSegments"]?
raise NotFoundException.new("Requested transcript does not exist")
end
# Extract user-friendly label for the current transcript
footer_language_menu = transcript_panel.dig?(
"footer", "transcriptFooterRenderer", "languageMenu", "sortFilterSubMenuRenderer", "subMenuItems"
)
if footer_language_menu
label = footer_language_menu.as_a.select(&.["selected"].as_bool)[0]["title"].as_s
else
label = language_code
end
# Extract transcript lines
initial_segments = segment_list["initialSegments"].as_a
lines = [] of TranscriptLine
initial_segments.each do |line|
if unpacked_line = line["transcriptSectionHeaderRenderer"]?
line_type = HeadingLine
else
unpacked_line = line["transcriptSegmentRenderer"]
line_type = RegularLine
end
start_ms = unpacked_line["startMs"].as_s.to_i.millisecond
end_ms = unpacked_line["endMs"].as_s.to_i.millisecond
text = extract_text(unpacked_line["snippet"]) || ""
lines << line_type.new(start_ms, end_ms, text)
end
return Transcript.new(
lines: lines,
language_code: language_code,
auto_generated: auto_generated,
label: label
)
end
# Converts transcript lines to a WebVTT file
#
# This is used within Invidious to replace subtitles
# as to workaround YouTube's rate-limited timedtext endpoint.
def to_vtt
settings_field = {
"Kind" => "captions",
"Language" => target_language,
"Language" => @language_code,
}
# Taken from Invidious::Videos::Captions::Metadata.timedtext_to_vtt()
vtt = WebVTT.build(settings_field) do |vtt|
lines.each do |line|
@lines.each do |line|
# Section headers are excluded from the VTT conversion as to
# match the regular captions returned from YouTube as much as possible
next if line.is_a? HeadingLine
vtt.cue(line.start_ms, line.end_ms, line.line)
end
end
return vtt
end
private def self.parse(initial_data : Hash(String, JSON::Any))
body = initial_data.dig("actions", 0, "updateEngagementPanelAction", "content", "transcriptRenderer",
"content", "transcriptSearchPanelRenderer", "body", "transcriptSegmentListRenderer",
"initialSegments").as_a
lines = [] of TranscriptLine
body.each do |line|
# Transcript section headers. They are not apart of the captions and as such we can safely skip them.
if line.as_h.has_key?("transcriptSectionHeaderRenderer")
next
end
line = line["transcriptSegmentRenderer"]
start_ms = line["startMs"].as_s.to_i.millisecond
end_ms = line["endMs"].as_s.to_i.millisecond
text = extract_text(line["snippet"]) || ""
lines << TranscriptLine.new(start_ms, end_ms, text)
end
return lines
end
end
end

View File

@ -30,13 +30,13 @@
<meta property="og:site_name" content="Invidious">
<meta property="og:url" content="<%= HOST_URL %>/channel/<%= ucid %>">
<meta property="og:title" content="<%= author %>">
<meta property="og:image" content="/ggpht<%= channel_profile_pic %>">
<meta property="og:image" content="<%= HOST_URL %>/ggpht<%= channel_profile_pic %>">
<meta property="og:description" content="<%= channel.description %>">
<meta name="twitter:card" content="summary">
<meta name="twitter:url" content="<%= HOST_URL %>/channel/<%= ucid %>">
<meta name="twitter:title" content="<%= author %>">
<meta name="twitter:description" content="<%= channel.description %>">
<meta name="twitter:image" content="/ggpht<%= channel_profile_pic %>">
<meta name="twitter:image" content="<%= HOST_URL %>/ggpht<%= channel_profile_pic %>">
<link rel="alternate" type="application/rss+xml" title="RSS" href="/feed/channel/<%= ucid %>" />
<%- end -%>

View File

@ -310,7 +310,7 @@
<div class="pure-control-group">
<label for="modified_source_code_url"><%= translate(locale, "adminprefs_modified_source_code_url_label") %></label>
<input name="modified_source_code_url" id="modified_source_code_url" type="input" <% if CONFIG.modified_source_code_url %>checked<% end %>>
<input name="modified_source_code_url" id="modified_source_code_url" type="url" value="<%= CONFIG.modified_source_code_url %>">
</div>
<% end %>

View File

@ -10,7 +10,7 @@
<meta property="og:site_name" content="<%= author %> | Invidious">
<meta property="og:url" content="<%= HOST_URL %>/watch?v=<%= video.id %>">
<meta property="og:title" content="<%= title %>">
<meta property="og:image" content="/vi/<%= video.id %>/maxres.jpg">
<meta property="og:image" content="<%= HOST_URL %>/vi/<%= video.id %>/maxres.jpg">
<meta property="og:description" content="<%= HTML.escape(video.short_description) %>">
<meta property="og:type" content="video.other">
<meta property="og:video:url" content="<%= HOST_URL %>/embed/<%= video.id %>">

View File

@ -5,9 +5,6 @@
module YoutubeAPI
extend self
private DEFAULT_API_KEY = "AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8"
private ANDROID_API_KEY = "AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w"
# For Android versions, see https://en.wikipedia.org/wiki/Android_version_history
private ANDROID_APP_VERSION = "19.14.42"
private ANDROID_USER_AGENT = "com.google.android.youtube/19.14.42 (Linux; U; Android 12; US) gzip"
@ -52,7 +49,6 @@ module YoutubeAPI
name: "WEB",
name_proto: "1",
version: "2.20240304.00.00",
api_key: DEFAULT_API_KEY,
screen: "WATCH_FULL_SCREEN",
os_name: "Windows",
os_version: WINDOWS_VERSION,
@ -62,7 +58,6 @@ module YoutubeAPI
name: "WEB_EMBEDDED_PLAYER",
name_proto: "56",
version: "1.20240303.00.00",
api_key: DEFAULT_API_KEY,
screen: "EMBED",
os_name: "Windows",
os_version: WINDOWS_VERSION,
@ -72,7 +67,6 @@ module YoutubeAPI
name: "MWEB",
name_proto: "2",
version: "2.20240304.08.00",
api_key: DEFAULT_API_KEY,
os_name: "Android",
os_version: ANDROID_VERSION,
platform: "MOBILE",
@ -81,7 +75,6 @@ module YoutubeAPI
name: "WEB",
name_proto: "1",
version: "2.20240304.00.00",
api_key: DEFAULT_API_KEY,
screen: "EMBED",
os_name: "Windows",
os_version: WINDOWS_VERSION,
@ -94,7 +87,6 @@ module YoutubeAPI
name: "ANDROID",
name_proto: "3",
version: ANDROID_APP_VERSION,
api_key: ANDROID_API_KEY,
android_sdk_version: ANDROID_SDK_VERSION,
user_agent: ANDROID_USER_AGENT,
os_name: "Android",
@ -105,13 +97,11 @@ module YoutubeAPI
name: "ANDROID_EMBEDDED_PLAYER",
name_proto: "55",
version: ANDROID_APP_VERSION,
api_key: "AIzaSyCjc_pVEDi4qsv5MtC2dMXzpIaDoRFLsxw",
},
ClientType::AndroidScreenEmbed => {
name: "ANDROID",
name_proto: "3",
version: ANDROID_APP_VERSION,
api_key: DEFAULT_API_KEY,
screen: "EMBED",
android_sdk_version: ANDROID_SDK_VERSION,
user_agent: ANDROID_USER_AGENT,
@ -123,7 +113,6 @@ module YoutubeAPI
name: "ANDROID_TESTSUITE",
name_proto: "30",
version: ANDROID_TS_APP_VERSION,
api_key: ANDROID_API_KEY,
android_sdk_version: ANDROID_SDK_VERSION,
user_agent: ANDROID_TS_USER_AGENT,
os_name: "Android",
@ -137,7 +126,6 @@ module YoutubeAPI
name: "IOS",
name_proto: "5",
version: IOS_APP_VERSION,
api_key: "AIzaSyB-63vPrdThhKuerbB2N_l7Kwwcxj6yUAc",
user_agent: IOS_USER_AGENT,
device_make: "Apple",
device_model: "iPhone14,5",
@ -149,7 +137,6 @@ module YoutubeAPI
name: "IOS_MESSAGES_EXTENSION",
name_proto: "66",
version: IOS_APP_VERSION,
api_key: DEFAULT_API_KEY,
user_agent: IOS_USER_AGENT,
device_make: "Apple",
device_model: "iPhone14,5",
@ -161,7 +148,6 @@ module YoutubeAPI
name: "IOS_MUSIC",
name_proto: "26",
version: "6.42",
api_key: "AIzaSyBAETezhkwP0ZWA02RsqT1zu78Fpt0bC_s",
user_agent: "com.google.ios.youtubemusic/6.42 (iPhone14,5; U; CPU iOS 17_4 like Mac OS X;)",
device_make: "Apple",
device_model: "iPhone14,5",
@ -176,13 +162,11 @@ module YoutubeAPI
name: "TVHTML5",
name_proto: "7",
version: "7.20240304.10.00",
api_key: DEFAULT_API_KEY,
},
ClientType::TvHtml5ScreenEmbed => {
name: "TVHTML5_SIMPLY_EMBEDDED_PLAYER",
name_proto: "85",
version: "2.0",
api_key: DEFAULT_API_KEY,
screen: "EMBED",
},
}
@ -237,11 +221,6 @@ module YoutubeAPI
HARDCODED_CLIENTS[@client_type][:version]
end
# :ditto:
def api_key : String
HARDCODED_CLIENTS[@client_type][:api_key]
end
# :ditto:
def screen : String
HARDCODED_CLIENTS[@client_type][:screen]? || ""
@ -606,7 +585,7 @@ module YoutubeAPI
client_config ||= DEFAULT_CLIENT_CONFIG
# Query parameters
url = "#{endpoint}?key=#{client_config.api_key}&prettyPrint=false"
url = "#{endpoint}?prettyPrint=false"
headers = HTTP::Headers{
"Content-Type" => "application/json; charset=UTF-8",