Compare commits

...

7 Commits

Author SHA1 Message Date
choelzl
dc1ed9ed4f
Merge 8953c105be into 3e17d04875 2024-08-26 11:18:07 +02:00
Samantaz Fox
3e17d04875
Release v2.20240825.1 2024-08-25 22:30:46 +02:00
syeopite
cec905e95e
Allow manual trigger of release-container build (#4877) 2024-08-25 19:55:52 +00:00
Samantaz Fox
80958aa0d8
Release v2.20240825 2024-08-25 21:25:48 +02:00
soraefir
8953c105be
Implemented requested changes and added 'auth_enforce_source' option 2023-07-13 12:48:18 +02:00
soraefir
4d14789e7b
Added login flow functions and fix cookies 2023-07-12 01:09:29 +02:00
soraefir
bee301a6f4
Implemented Oauth 2023-07-12 00:38:03 +02:00
10 changed files with 414 additions and 50 deletions

View File

@ -1,6 +1,7 @@
name: Build and release container
on:
workflow_dispatch:
push:
tags:
- "v*"

View File

@ -1,6 +1,175 @@
# CHANGELOG
## 2024-04-26
## v2.20240825.1 (2024-08-25)
Add patch component to be [semver] compliant and make github actions happy.
[semver]: https://semver.org/
### Full list of pull requests merged since the last release (newest first)
Allow manual trigger of release-container build (#4877, thanks @syeopite)
## v2.20240825.0 (2024-08-25)
### New features & important changes
#### For users
* The search bar now has a button that you can click!
* Youtube URLs can be pasted directly in the search bar. Prepend search query with a
backslash (`\`) to disable that feature (useful if you need to search for a video whose
title contains some youtube URL).
* On the channel page the "streams" tab can be sorted by either: "newest", "oldest" or "popular"
* Lots of translations have been updated (thanks to our contributors on Weblate!)
* Videos embedded in local HTML files (e.g: a webpage saved from a blog) can now be played
#### For instance owners
* Invidious now has the ability to provide a `po_token` and `visitordata` to Youtube in order to
circumvent current Youtube restrictions.
* Invidious can use an (optional) external signature server like [inv_sig_helper]. Please note that
some videos can't be played without that signature server.
* The Helm charts were moved to a separate repo: https://github.com/iv-org/invidious-helm-chart
* We have changed how containers are released: the `latest` tag now tracks tagged releases, whereas
the `master` tag tracks the most recent commits of the `master` branch ("nightly" builds).
[inv_sig_helper]: https://github.com/iv-org/inv_sig_helper
#### For developpers
* The versions of Crystal that we test in CI/CD are now: `1.9.2`, `1.10.1`, `1.11.2`, `1.12.1`.
Please note that due to a bug in the `libxml` bindings (See [#4256]), versions prior to `1.10.0`
are not recommended to use.
* Thanks to @syeopite, the code is now [ameba] compliant.
* Ameba is part of our CI/CD pipeline, and its rules will be enforced in future PRs.
* The transcript code has been rewritten to permit transcripts as a feature rather than being
only a workaround for captions. Trancripts feature is coming soon!
* Various fixes regarding the logic interacting with Youtube
* The `sort_by` parameter can be used on the `/api/v1/channels/{id}/streams` endpoint. Accepted
values are: "newest", "oldest" and "popular"
[ameba]: https://github.com/crystal-ameba/ameba
[#4256]: https://github.com/iv-org/invidious/issues/4256
### Bugs fixed
#### User-side
* Channels: fixed broken "subscribers" and "views" counters
* Watch page: playback position is reset at the end of a video, so that the next time this video
is watched, it will start from the beginning rather than 15 seconds before the end
* Watch page: the items in the "add to playlist" drop down are now sorted alphabetically
* Videos: the "genre" URL is now always pointing to a valid webpage
* Playlists: Fixed `Could not parse N episodes` error on podcast playlists
* All external links should now have the [`rel`] attibute set to `noreferrer noopener` for
increased privacy.
* Preferences: Fixed the admin-only "modified source code" input being ignored
* Watch/channel pages: use the full image URL in `og:image` and `twitter:image` meta tags
[`rel`]: https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel
#### API
* fixed the `local` parameter not applying to `formatStreams` on `/api/v1/videos/{id}`
* fixed an `Index out of bounds` error hapenning when a playlist had no videos
* fixed duplicated query parameters in proxied video URLs
* Return actual video height/width/fps rather than hard coded values
* Fixed the `/api/v1/popular` endpoint not returning a proper error code/message when the
popular page/endpoint are disabled.
### Full list of pull requests merged since the last release (newest first)
* HTML: Sort playlists alphabetically in watch page drop down ([#4853], by @SamantazFox)
* Videos: Fix XSS vulnerability in description/comments ([#4852], thanks _anonymous_)
* YtAPI: Bump client versions ([#4849], by @SamantazFox)
* SigHelper: Fix inverted time comparison in 'check_update' ([#4845], by @SamantazFox)
* Storyboards: Various fixes and code cleaning ([#4153], by SamantazFox)
* Fix lint errors introduced in #4146 and #4295 ([#4876], thanks @syeopite)
* Search: Add support for Youtube URLs ([#4146], by @SamantazFox)
* Channel: Render age restricted channels ([#4295], thanks @ChunkyProgrammer)
* Ameba: Miscellaneous fixes ([#4807], thanks @syeopite)
* API: Proxy formatStreams URLs too ([#4859], thanks @colinleroy)
* UI: Add search button to search bar ([#4706], thanks @thansk)
* Add ability to set po_token and visitordata ID ([#4789], thanks @unixfox)
* Add support for an external signature server ([#4772], by @SamantazFox)
* Ameba: Fix Naming/VariableNames ([#4790], thanks @syeopite)
* Translations update from Hosted Weblate ([#4659])
* Ameba: Fix Lint/UselessAssign ([#4795], thanks @syeopite)
* HTML: Add rel="noreferrer noopener" to external links ([#4667], thanks @ulmemxpoc)
* Remove unused methods in Invidious::LogHandler ([#4812], thanks @syeopite)
* Ameba: Fix Lint/NotNilAfterNoBang ([#4796], thanks @syeopite)
* Ameba: Fix unused argument Lint warnings ([#4805], thanks @syeopite)
* Ameba: i18next.cr fixes ([#4806], thanks @syeopite)
* Ameba: Disable rules ([#4792], thanks @syeopite)
* Channel: parse subscriber count and channel banner ([#4785], thanks @ChunkyProgrammer)
* Player: Fix playback position of already watched videos ([#4731], thanks @Fijxu)
* Videos: Fix genre url being unusable ([#4717], thanks @meatball133)
* API: Fix out of bound error on empty playlists ([#4696], thanks @Fijxu)
* Handle playlists cataloged as Podcast ([#4695], thanks @Fijxu)
* API: Fix duplicated query parameters in proxied video URLs ([#4587], thanks @absidue)
* API: Return actual stream height, width and fps ([#4586], thanks @absidue)
* Preferences: Fix handling of modified source code URL ([#4437], thanks @nooptek)
* API: Fix URL for vtt subtitles ([#4221], thanks @karelrooted)
* Channels: Add sort options to streams ([#4224], thanks @src-tinkerer)
* API: Fix error code for disabled popular endpoint ([#4296], thanks @iBicha)
* Allow embedding videos in local HTML files ([#4450], thanks @tomasz1986)
* CI: Bump Crystal version matrix ([#4654], by @SamantazFox)
* YtAPI: Remove API keys like official clients ([#4655], by @SamantazFox)
* HTML: Use full URL in the og:image property ([#4675], thanks @Fijxu)
* Rewrite transcript logic to be more generic ([#4747], thanks @syeopite)
* CI: Run Ameba ([#4753], thanks @syeopite)
* CI: Add release based containers ([#4763], thanks @syeopite)
* move helm chart to a dedicated github repository ([#4711], thanks @unixfox)
[#4146]: https://github.com/iv-org/invidious/pull/4146
[#4153]: https://github.com/iv-org/invidious/pull/4153
[#4221]: https://github.com/iv-org/invidious/pull/4221
[#4224]: https://github.com/iv-org/invidious/pull/4224
[#4295]: https://github.com/iv-org/invidious/pull/4295
[#4296]: https://github.com/iv-org/invidious/pull/4296
[#4437]: https://github.com/iv-org/invidious/pull/4437
[#4450]: https://github.com/iv-org/invidious/pull/4450
[#4586]: https://github.com/iv-org/invidious/pull/4586
[#4587]: https://github.com/iv-org/invidious/pull/4587
[#4654]: https://github.com/iv-org/invidious/pull/4654
[#4655]: https://github.com/iv-org/invidious/pull/4655
[#4659]: https://github.com/iv-org/invidious/pull/4659
[#4667]: https://github.com/iv-org/invidious/pull/4667
[#4675]: https://github.com/iv-org/invidious/pull/4675
[#4695]: https://github.com/iv-org/invidious/pull/4695
[#4696]: https://github.com/iv-org/invidious/pull/4696
[#4706]: https://github.com/iv-org/invidious/pull/4706
[#4711]: https://github.com/iv-org/invidious/pull/4711
[#4717]: https://github.com/iv-org/invidious/pull/4717
[#4731]: https://github.com/iv-org/invidious/pull/4731
[#4747]: https://github.com/iv-org/invidious/pull/4747
[#4753]: https://github.com/iv-org/invidious/pull/4753
[#4763]: https://github.com/iv-org/invidious/pull/4763
[#4772]: https://github.com/iv-org/invidious/pull/4772
[#4785]: https://github.com/iv-org/invidious/pull/4785
[#4789]: https://github.com/iv-org/invidious/pull/4789
[#4790]: https://github.com/iv-org/invidious/pull/4790
[#4792]: https://github.com/iv-org/invidious/pull/4792
[#4795]: https://github.com/iv-org/invidious/pull/4795
[#4796]: https://github.com/iv-org/invidious/pull/4796
[#4805]: https://github.com/iv-org/invidious/pull/4805
[#4806]: https://github.com/iv-org/invidious/pull/4806
[#4807]: https://github.com/iv-org/invidious/pull/4807
[#4812]: https://github.com/iv-org/invidious/pull/4812
[#4845]: https://github.com/iv-org/invidious/pull/4845
[#4849]: https://github.com/iv-org/invidious/pull/4849
[#4852]: https://github.com/iv-org/invidious/pull/4852
[#4853]: https://github.com/iv-org/invidious/pull/4853
[#4859]: https://github.com/iv-org/invidious/pull/4859
[#4876]: https://github.com/iv-org/invidious/pull/4876
## v2.20240427 (2024-04-27)
Major bug fixes:
* Videos: Use android test suite client (#4650, thanks @SamantazFox)

View File

@ -323,6 +323,40 @@ https_only: false
##
#enable_user_notifications: true
##
## List of Enabled Authentication Backend
## If not provided falls back to default
##
## Supported Values:
## - invidious
## - oauth
## - ldap (Not implemented !)
## - saml (Not implemented !)
##
## Default: ["invidious","oauth"]
##
# auth_type: ["oauth"]
##
## OAuth Configuration
##
## Notes:
## - Supports multiple OAuth backends
## - Requires external_port and domain to be configured
##
## Default: []
##
# oauth:
# example:
# host: oauth.example.net
# field : email
# auth_uri: /oauth/authorize/
# token_uri: /oauth/token/
# info_uri: https://api.example.net/oauth/userinfo/
# client_id: CLIENT_ID
# client_secret: CLIENT_SECRET
# -----------------------------
# Background jobs
# -----------------------------

View File

@ -8,6 +8,18 @@ struct DBConfig
property dbname : String
end
struct OAuthConfig
include YAML::Serializable
property host : String
property field : String = "email"
property auth_uri : String
property token_uri : String
property info_uri : String
property client_id : String
property client_secret : String
end
struct ConfigPreferences
include YAML::Serializable
@ -137,6 +149,10 @@ class Config
# poToken for passing bot attestation
property po_token : String? = nil
property auth_type : Array(String) = ["invidious", "oauth"]
property auth_enforce_source : Bool = true
property oauth = {} of String => OAuthConfig
# Saved cookies in "name1=value1; name2=value2..." format
@[YAML::Field(converter: Preferences::StringToCookies)]
property cookies : HTTP::Cookies = HTTP::Cookies.new
@ -159,6 +175,14 @@ class Config
end
end
def auth_oauth_enabled?
return (@auth_type.find(&.== "oauth") && @oauth.size > 0)
end
def auth_internal_enabled?
return (@auth_type.find(&.== "invidious"))
end
def self.load
# Load config from file or YAML string env var
env_config_file = "INVIDIOUS_CONFIG_FILE"

View File

@ -0,0 +1,53 @@
require "oauth2"
module Invidious::OAuthHelper
extend self
def get_provider(key)
if provider = CONFIG.oauth[key]?
provider
else
raise Exception.new("Invalid OAuth Endpoint: " + key)
end
end
def make_client(key)
if HOST_URL == ""
raise Exception.new("Missing domain and port configuration")
end
provider = get_provider(key)
redirect_uri = "#{HOST_URL}/login/oauth/#{key}"
OAuth2::Client.new(
provider.host,
provider.client_id,
provider.client_secret,
authorize_uri: provider.auth_uri,
token_uri: provider.token_uri,
redirect_uri: redirect_uri
)
end
def get_uri_host_pair(host, url)
if (url.starts_with?(/https*\:\/\//))
uri = URI.parse url
[uri.host || host, uri.path || "/"]
else
[host, url]
end
end
def get_info(key, token)
provider = self.get_provider(key)
uri_host_pair = self.get_uri_host_pair(provider.host, provider.info_uri)
client = HTTP::Client.new(uri_host_pair[0], tls: true)
token.authenticate(client)
response = client.get uri_host_pair[1]
client.close
response.body
end
def info_field(key, token)
info = JSON.parse(self.get_info(key, token))
info[self.get_provider(key).field].as_s?
end
end

View File

@ -3,11 +3,10 @@
module Invidious::Routes::Login
def self.login_page(env)
locale = env.get("preferences").as(Preferences).locale
referer = get_referer(env, "/feed/subscriptions")
user = env.get? "user"
referer = get_referer(env, "/feed/subscriptions")
return env.redirect referer if user
if !CONFIG.login_enabled
@ -19,7 +18,13 @@ module Invidious::Routes::Login
captcha = nil
account_type = env.params.query["type"]?
account_type ||= "invidious"
account_type ||= ""
if CONFIG.auth_type.size == 0
return error_template(401, "No authentication backend enabled.")
elsif CONFIG.auth_type.find(&.== account_type).nil? && CONFIG.auth_type.size == 1
account_type = CONFIG.auth_type[0]
end
captcha_type = env.params.query["captcha"]?
captcha_type ||= "image"
@ -27,9 +32,38 @@ module Invidious::Routes::Login
templated "user/login"
end
def self.login_oauth(env)
locale = env.get("preferences").as(Preferences).locale
referer = get_referer(env, "/feed/subscriptions")
authorization_code = env.params.query["code"]?
provider_k = env.params.url["provider"]
if authorization_code.nil?
return error_template(403, "Missing Authorization Code")
end
begin
token = OAuthHelper.make_client(provider_k).get_access_token_using_authorization_code(authorization_code)
if email = OAuthHelper.info_field(provider_k, token)
if user = Invidious::Database::Users.select(email: email)
if CONFIG.auth_enforce_source && user.password != ("oauth:" + provider_k)
return error_template(401, "Wrong provider")
else
user_flow_existing(env, email)
end
else
user_flow_new(env, email, nil, "oauth:" + provider_k)
end
end
rescue ex
return error_template(500, "Internal Error")
end
env.redirect referer
end
def self.login(env)
locale = env.get("preferences").as(Preferences).locale
referer = get_referer(env, "/feed/subscriptions")
if !CONFIG.login_enabled
@ -41,9 +75,22 @@ module Invidious::Routes::Login
password = env.params.body["password"]?
account_type = env.params.query["type"]?
account_type ||= "invidious"
account_type ||= ""
if CONFIG.auth_type.size == 0
return error_template(401, "No authentication backend enabled.")
elsif CONFIG.auth_type.find(&.== account_type).nil? && CONFIG.auth_type.size == 1
account_type = CONFIG.auth_type[0]
end
case account_type
when "oauth"
provider_k = env.params.body["provider"]
env.redirect OAuthHelper.make_client(provider_k).get_authorize_uri("openid email profile")
when "saml"
return error_template(501, "Not implemented")
when "ldap"
return error_template(501, "Not implemented")
when "invidious"
if email.nil? || email.empty?
return error_template(401, "User ID is a required field")
@ -53,24 +100,14 @@ module Invidious::Routes::Login
return error_template(401, "Password is a required field")
end
user = Invidious::Database::Users.select(email: email)
if user
if Crypto::Bcrypt::Password.new(user.password.not_nil!).verify(password.byte_slice(0, 55))
sid = Base64.urlsafe_encode(Random::Secure.random_bytes(32))
Invidious::Database::SessionIDs.insert(sid, email)
env.response.cookies["SID"] = Invidious::User::Cookies.sid(CONFIG.domain, sid)
if user = Invidious::Database::Users.select(email: email)
if user.password.not_nil!.starts_with? "oauth"
return error_template(401, "Wrong provider")
elsif Crypto::Bcrypt::Password.new(user.password.not_nil!).verify(password.byte_slice(0, 55))
user_flow_existing(env, email)
else
return error_template(401, "Wrong username or password")
end
# Since this user has already registered, we don't want to overwrite their preferences
if env.request.cookies["PREFS"]?
cookie = env.request.cookies["PREFS"]
cookie.expires = Time.utc(1990, 1, 1)
env.response.cookies << cookie
end
else
if !CONFIG.registration_enabled
return error_template(400, "Registration has been disabled by administrator.")
@ -147,32 +184,7 @@ module Invidious::Routes::Login
end
end
end
sid = Base64.urlsafe_encode(Random::Secure.random_bytes(32))
user, sid = create_user(sid, email, password)
if language_header = env.request.headers["Accept-Language"]?
if language = ANG.language_negotiator.best(language_header, LOCALES.keys)
user.preferences.locale = language.header
end
end
Invidious::Database::Users.insert(user)
Invidious::Database::SessionIDs.insert(sid, email)
view_name = "subscriptions_#{sha256(user.email)}"
PG_DB.exec("CREATE MATERIALIZED VIEW #{view_name} AS #{MATERIALIZED_VIEW_SQL.call(user.email)}")
env.response.cookies["SID"] = Invidious::User::Cookies.sid(CONFIG.domain, sid)
if env.request.cookies["PREFS"]?
user.preferences = env.get("preferences").as(Preferences)
Invidious::Database::Users.update_preferences(user)
cookie = env.request.cookies["PREFS"]
cookie.expires = Time.utc(1990, 1, 1)
env.response.cookies << cookie
end
user_flow_new(env, email, password, "internal")
end
env.redirect referer
@ -211,4 +223,49 @@ module Invidious::Routes::Login
env.redirect referer
end
def self.user_flow_existing(env, email)
sid = Base64.urlsafe_encode(Random::Secure.random_bytes(32))
Invidious::Database::SessionIDs.insert(sid, email)
env.response.cookies["SID"] = Invidious::User::Cookies.sid(CONFIG.domain, sid)
# Since this user has already registered, we don't want to overwrite their preferences
if env.request.cookies["PREFS"]?
cookie = env.request.cookies["PREFS"]
cookie.expires = Time.utc(1990, 1, 1)
env.response.cookies << cookie
end
end
def self.user_flow_new(env, email, password, provider)
sid = Base64.urlsafe_encode(Random::Secure.random_bytes(32))
if provider == "internal"
user, sid = create_internal_user(sid, email, password)
else
user, sid = create_user(sid, email, provider)
end
if language_header = env.request.headers["Accept-Language"]?
if language = ANG.language_negotiator.best(language_header, LOCALES.keys)
user.preferences.locale = language.header
end
end
Invidious::Database::Users.insert(user)
Invidious::Database::SessionIDs.insert(sid, email)
view_name = "subscriptions_#{sha256(user.email)}"
PG_DB.exec("CREATE MATERIALIZED VIEW #{view_name} AS #{MATERIALIZED_VIEW_SQL.call(user.email)}")
env.response.cookies["SID"] = Invidious::User::Cookies.sid(CONFIG.domain, sid)
if env.request.cookies["PREFS"]?
user.preferences = env.get("preferences").as(Preferences)
Invidious::Database::Users.update_preferences(user)
cookie = env.request.cookies["PREFS"]
cookie.expires = Time.utc(1990, 1, 1)
env.response.cookies << cookie
end
end
end

View File

@ -55,6 +55,7 @@ module Invidious::Routing
def register_user_routes
# User login/out
get "/login", Routes::Login, :login_page
get "/login/oauth/:provider", Routes::Login, :login_oauth
post "/login", Routes::Login, :login
post "/signout", Routes::Login, :signout

View File

@ -14,6 +14,7 @@ struct Invidious::User
return HTTP::Cookie.new(
name: "SID",
domain: domain,
path: "/",
value: sid,
expires: Time.utc + 2.years,
secure: SECURE,
@ -28,6 +29,7 @@ struct Invidious::User
return HTTP::Cookie.new(
name: "PREFS",
domain: domain,
path: "/",
value: URI.encode_www_form(preferences.to_json),
expires: Time.utc + 2.years,
secure: SECURE,

View File

@ -4,7 +4,6 @@ require "crypto/bcrypt/password"
MATERIALIZED_VIEW_SQL = ->(email : String) { "SELECT cv.* FROM channel_videos cv WHERE EXISTS (SELECT subscriptions FROM users u WHERE cv.ucid = ANY (u.subscriptions) AND u.email = E'#{email.gsub({'\'' => "\\'", '\\' => "\\\\"})}') ORDER BY published DESC" }
def create_user(sid, email, password)
password = Crypto::Bcrypt::Password.create(password, cost: 10)
token = Base64.urlsafe_encode(Random::Secure.random_bytes(32))
user = Invidious::User.new({
@ -13,7 +12,7 @@ def create_user(sid, email, password)
subscriptions: [] of String,
email: email,
preferences: Preferences.new(CONFIG.default_user_preferences.to_tuple),
password: password.to_s,
password: password,
token: token,
watched: [] of String,
feed_needs_update: true,
@ -22,6 +21,11 @@ def create_user(sid, email, password)
return user, sid
end
def create_internal_user(sid, email, password)
password = Crypto::Bcrypt::Password.create(password.not_nil!, cost: 10)
create_user(sid, email, password.to_s)
end
def get_subscription_feed(user, max_results = 40, page = 1)
limit = max_results.clamp(0, MAX_ITEMS_PER_PAGE)
offset = (page - 1) * limit

View File

@ -7,7 +7,18 @@
<div class="pure-u-1 pure-u-lg-3-5">
<div class="h-box">
<% case account_type when %>
<% else # "invidious" %>
<% when "oauth" %>
<form class="pure-form pure-form-stacked" action="/login?referer=<%= URI.encode_www_form(referer) %>&type=oauth" method="post">
<fieldset>
<select name="provider" id="provider">
<% CONFIG.oauth.each_key do |key| %>
<option value="<%= key %>"><%= key %></option>
<% end %>
</select>
<button type="submit" class="pure-button pure-button-primary"><%= translate(locale, "Sign In via OAuth") %></button>
</fieldset>
</form>
<% when "invidious" %>
<form class="pure-form pure-form-stacked" action="/login?referer=<%= URI.encode_www_form(referer) %>&type=invidious" method="post">
<fieldset>
<% if email %>
@ -70,6 +81,14 @@
<% end %>
</fieldset>
</form>
<% else %>
<% if CONFIG.auth_internal_enabled? %>
<a class="pure-button pure-button-secondary" href="/login?referer=<%= URI.encode_www_form(referer) %>&type=invidious">Internal</a>
<% end %>
<% if CONFIG.auth_oauth_enabled? %>
<a class="pure-button pure-button-secondary" href="/login?referer=<%= URI.encode_www_form(referer) %>&type=oauth">OAuth</a>
<% end %>
<label></label>
<% end %>
</div>
</div>