Compare commits

...

5 Commits

Author SHA1 Message Date
sophie [⛧-440729]
5ee20fdcea
Merge 78768f18ae into 3e17d04875 2024-08-25 22:31:28 +02:00
Samantaz Fox
3e17d04875
Release v2.20240825.1 2024-08-25 22:30:46 +02:00
syeopite
cec905e95e
Allow manual trigger of release-container build (#4877) 2024-08-25 19:55:52 +00:00
Samantaz Fox
80958aa0d8
Release v2.20240825 2024-08-25 21:25:48 +02:00
Sophie Tauchert
78768f18ae
Batch user notifications together 2024-03-16 01:23:13 +01:00
7 changed files with 259 additions and 34 deletions

View File

@ -1,6 +1,7 @@
name: Build and release container name: Build and release container
on: on:
workflow_dispatch:
push: push:
tags: tags:
- "v*" - "v*"

View File

@ -1,6 +1,175 @@
# CHANGELOG # CHANGELOG
## 2024-04-26 ## v2.20240825.1 (2024-08-25)
Add patch component to be [semver] compliant and make github actions happy.
[semver]: https://semver.org/
### Full list of pull requests merged since the last release (newest first)
Allow manual trigger of release-container build (#4877, thanks @syeopite)
## v2.20240825.0 (2024-08-25)
### New features & important changes
#### For users
* The search bar now has a button that you can click!
* Youtube URLs can be pasted directly in the search bar. Prepend search query with a
backslash (`\`) to disable that feature (useful if you need to search for a video whose
title contains some youtube URL).
* On the channel page the "streams" tab can be sorted by either: "newest", "oldest" or "popular"
* Lots of translations have been updated (thanks to our contributors on Weblate!)
* Videos embedded in local HTML files (e.g: a webpage saved from a blog) can now be played
#### For instance owners
* Invidious now has the ability to provide a `po_token` and `visitordata` to Youtube in order to
circumvent current Youtube restrictions.
* Invidious can use an (optional) external signature server like [inv_sig_helper]. Please note that
some videos can't be played without that signature server.
* The Helm charts were moved to a separate repo: https://github.com/iv-org/invidious-helm-chart
* We have changed how containers are released: the `latest` tag now tracks tagged releases, whereas
the `master` tag tracks the most recent commits of the `master` branch ("nightly" builds).
[inv_sig_helper]: https://github.com/iv-org/inv_sig_helper
#### For developpers
* The versions of Crystal that we test in CI/CD are now: `1.9.2`, `1.10.1`, `1.11.2`, `1.12.1`.
Please note that due to a bug in the `libxml` bindings (See [#4256]), versions prior to `1.10.0`
are not recommended to use.
* Thanks to @syeopite, the code is now [ameba] compliant.
* Ameba is part of our CI/CD pipeline, and its rules will be enforced in future PRs.
* The transcript code has been rewritten to permit transcripts as a feature rather than being
only a workaround for captions. Trancripts feature is coming soon!
* Various fixes regarding the logic interacting with Youtube
* The `sort_by` parameter can be used on the `/api/v1/channels/{id}/streams` endpoint. Accepted
values are: "newest", "oldest" and "popular"
[ameba]: https://github.com/crystal-ameba/ameba
[#4256]: https://github.com/iv-org/invidious/issues/4256
### Bugs fixed
#### User-side
* Channels: fixed broken "subscribers" and "views" counters
* Watch page: playback position is reset at the end of a video, so that the next time this video
is watched, it will start from the beginning rather than 15 seconds before the end
* Watch page: the items in the "add to playlist" drop down are now sorted alphabetically
* Videos: the "genre" URL is now always pointing to a valid webpage
* Playlists: Fixed `Could not parse N episodes` error on podcast playlists
* All external links should now have the [`rel`] attibute set to `noreferrer noopener` for
increased privacy.
* Preferences: Fixed the admin-only "modified source code" input being ignored
* Watch/channel pages: use the full image URL in `og:image` and `twitter:image` meta tags
[`rel`]: https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel
#### API
* fixed the `local` parameter not applying to `formatStreams` on `/api/v1/videos/{id}`
* fixed an `Index out of bounds` error hapenning when a playlist had no videos
* fixed duplicated query parameters in proxied video URLs
* Return actual video height/width/fps rather than hard coded values
* Fixed the `/api/v1/popular` endpoint not returning a proper error code/message when the
popular page/endpoint are disabled.
### Full list of pull requests merged since the last release (newest first)
* HTML: Sort playlists alphabetically in watch page drop down ([#4853], by @SamantazFox)
* Videos: Fix XSS vulnerability in description/comments ([#4852], thanks _anonymous_)
* YtAPI: Bump client versions ([#4849], by @SamantazFox)
* SigHelper: Fix inverted time comparison in 'check_update' ([#4845], by @SamantazFox)
* Storyboards: Various fixes and code cleaning ([#4153], by SamantazFox)
* Fix lint errors introduced in #4146 and #4295 ([#4876], thanks @syeopite)
* Search: Add support for Youtube URLs ([#4146], by @SamantazFox)
* Channel: Render age restricted channels ([#4295], thanks @ChunkyProgrammer)
* Ameba: Miscellaneous fixes ([#4807], thanks @syeopite)
* API: Proxy formatStreams URLs too ([#4859], thanks @colinleroy)
* UI: Add search button to search bar ([#4706], thanks @thansk)
* Add ability to set po_token and visitordata ID ([#4789], thanks @unixfox)
* Add support for an external signature server ([#4772], by @SamantazFox)
* Ameba: Fix Naming/VariableNames ([#4790], thanks @syeopite)
* Translations update from Hosted Weblate ([#4659])
* Ameba: Fix Lint/UselessAssign ([#4795], thanks @syeopite)
* HTML: Add rel="noreferrer noopener" to external links ([#4667], thanks @ulmemxpoc)
* Remove unused methods in Invidious::LogHandler ([#4812], thanks @syeopite)
* Ameba: Fix Lint/NotNilAfterNoBang ([#4796], thanks @syeopite)
* Ameba: Fix unused argument Lint warnings ([#4805], thanks @syeopite)
* Ameba: i18next.cr fixes ([#4806], thanks @syeopite)
* Ameba: Disable rules ([#4792], thanks @syeopite)
* Channel: parse subscriber count and channel banner ([#4785], thanks @ChunkyProgrammer)
* Player: Fix playback position of already watched videos ([#4731], thanks @Fijxu)
* Videos: Fix genre url being unusable ([#4717], thanks @meatball133)
* API: Fix out of bound error on empty playlists ([#4696], thanks @Fijxu)
* Handle playlists cataloged as Podcast ([#4695], thanks @Fijxu)
* API: Fix duplicated query parameters in proxied video URLs ([#4587], thanks @absidue)
* API: Return actual stream height, width and fps ([#4586], thanks @absidue)
* Preferences: Fix handling of modified source code URL ([#4437], thanks @nooptek)
* API: Fix URL for vtt subtitles ([#4221], thanks @karelrooted)
* Channels: Add sort options to streams ([#4224], thanks @src-tinkerer)
* API: Fix error code for disabled popular endpoint ([#4296], thanks @iBicha)
* Allow embedding videos in local HTML files ([#4450], thanks @tomasz1986)
* CI: Bump Crystal version matrix ([#4654], by @SamantazFox)
* YtAPI: Remove API keys like official clients ([#4655], by @SamantazFox)
* HTML: Use full URL in the og:image property ([#4675], thanks @Fijxu)
* Rewrite transcript logic to be more generic ([#4747], thanks @syeopite)
* CI: Run Ameba ([#4753], thanks @syeopite)
* CI: Add release based containers ([#4763], thanks @syeopite)
* move helm chart to a dedicated github repository ([#4711], thanks @unixfox)
[#4146]: https://github.com/iv-org/invidious/pull/4146
[#4153]: https://github.com/iv-org/invidious/pull/4153
[#4221]: https://github.com/iv-org/invidious/pull/4221
[#4224]: https://github.com/iv-org/invidious/pull/4224
[#4295]: https://github.com/iv-org/invidious/pull/4295
[#4296]: https://github.com/iv-org/invidious/pull/4296
[#4437]: https://github.com/iv-org/invidious/pull/4437
[#4450]: https://github.com/iv-org/invidious/pull/4450
[#4586]: https://github.com/iv-org/invidious/pull/4586
[#4587]: https://github.com/iv-org/invidious/pull/4587
[#4654]: https://github.com/iv-org/invidious/pull/4654
[#4655]: https://github.com/iv-org/invidious/pull/4655
[#4659]: https://github.com/iv-org/invidious/pull/4659
[#4667]: https://github.com/iv-org/invidious/pull/4667
[#4675]: https://github.com/iv-org/invidious/pull/4675
[#4695]: https://github.com/iv-org/invidious/pull/4695
[#4696]: https://github.com/iv-org/invidious/pull/4696
[#4706]: https://github.com/iv-org/invidious/pull/4706
[#4711]: https://github.com/iv-org/invidious/pull/4711
[#4717]: https://github.com/iv-org/invidious/pull/4717
[#4731]: https://github.com/iv-org/invidious/pull/4731
[#4747]: https://github.com/iv-org/invidious/pull/4747
[#4753]: https://github.com/iv-org/invidious/pull/4753
[#4763]: https://github.com/iv-org/invidious/pull/4763
[#4772]: https://github.com/iv-org/invidious/pull/4772
[#4785]: https://github.com/iv-org/invidious/pull/4785
[#4789]: https://github.com/iv-org/invidious/pull/4789
[#4790]: https://github.com/iv-org/invidious/pull/4790
[#4792]: https://github.com/iv-org/invidious/pull/4792
[#4795]: https://github.com/iv-org/invidious/pull/4795
[#4796]: https://github.com/iv-org/invidious/pull/4796
[#4805]: https://github.com/iv-org/invidious/pull/4805
[#4806]: https://github.com/iv-org/invidious/pull/4806
[#4807]: https://github.com/iv-org/invidious/pull/4807
[#4812]: https://github.com/iv-org/invidious/pull/4812
[#4845]: https://github.com/iv-org/invidious/pull/4845
[#4849]: https://github.com/iv-org/invidious/pull/4849
[#4852]: https://github.com/iv-org/invidious/pull/4852
[#4853]: https://github.com/iv-org/invidious/pull/4853
[#4859]: https://github.com/iv-org/invidious/pull/4859
[#4876]: https://github.com/iv-org/invidious/pull/4876
## v2.20240427 (2024-04-27)
Major bug fixes: Major bug fixes:
* Videos: Use android test suite client (#4650, thanks @SamantazFox) * Videos: Use android test suite client (#4650, thanks @SamantazFox)

View File

@ -184,8 +184,9 @@ if CONFIG.popular_enabled
Invidious::Jobs.register Invidious::Jobs::PullPopularVideosJob.new(PG_DB) Invidious::Jobs.register Invidious::Jobs::PullPopularVideosJob.new(PG_DB)
end end
CONNECTION_CHANNEL = ::Channel({Bool, ::Channel(PQ::Notification)}).new(32) NOTIFICATION_CHANNEL = ::Channel(VideoNotification).new(32)
Invidious::Jobs.register Invidious::Jobs::NotificationJob.new(CONNECTION_CHANNEL, CONFIG.database_url) CONNECTION_CHANNEL = ::Channel({Bool, ::Channel(PQ::Notification)}).new(32)
Invidious::Jobs.register Invidious::Jobs::NotificationJob.new(NOTIFICATION_CHANNEL, CONNECTION_CHANNEL, CONFIG.database_url)
Invidious::Jobs.register Invidious::Jobs::ClearExpiredItemsJob.new Invidious::Jobs.register Invidious::Jobs::ClearExpiredItemsJob.new

View File

@ -249,11 +249,7 @@ def fetch_channel(ucid, pull_all_videos : Bool)
if was_insert if was_insert
LOGGER.trace("fetch_channel: #{ucid} : video #{video_id} : Inserted, updating subscriptions") LOGGER.trace("fetch_channel: #{ucid} : video #{video_id} : Inserted, updating subscriptions")
if CONFIG.enable_user_notifications NOTIFICATION_CHANNEL.send(VideoNotification.from_video(video))
Invidious::Database::Users.add_notification(video)
else
Invidious::Database::Users.feed_needs_update(video)
end
else else
LOGGER.trace("fetch_channel: #{ucid} : video #{video_id} : Updated") LOGGER.trace("fetch_channel: #{ucid} : video #{video_id} : Updated")
end end
@ -285,11 +281,7 @@ def fetch_channel(ucid, pull_all_videos : Bool)
if Time.utc - video.published > 1.minute if Time.utc - video.published > 1.minute
was_insert = Invidious::Database::ChannelVideos.insert(video) was_insert = Invidious::Database::ChannelVideos.insert(video)
if was_insert if was_insert
if CONFIG.enable_user_notifications NOTIFICATION_CHANNEL.send(VideoNotification.from_video(video))
Invidious::Database::Users.add_notification(video)
else
Invidious::Database::Users.feed_needs_update(video)
end
end end
end end
end end

View File

@ -119,15 +119,15 @@ module Invidious::Database::Users
# Update (notifs) # Update (notifs)
# ------------------- # -------------------
def add_notification(video : ChannelVideo) def add_multiple_notifications(channel_id : String, video_ids : Array(String))
request = <<-SQL request = <<-SQL
UPDATE users UPDATE users
SET notifications = array_append(notifications, $1), SET notifications = array_cat(notifications, $1),
feed_needs_update = true feed_needs_update = true
WHERE $2 = ANY(subscriptions) WHERE $2 = ANY(subscriptions)
SQL SQL
PG_DB.exec(request, video.id, video.ucid) PG_DB.exec(request, video_ids, channel_id)
end end
def remove_notification(user : User, vid : String) def remove_notification(user : User, vid : String)
@ -154,14 +154,14 @@ module Invidious::Database::Users
# Update (misc) # Update (misc)
# ------------------- # -------------------
def feed_needs_update(video : ChannelVideo) def feed_needs_update(channel_id : String)
request = <<-SQL request = <<-SQL
UPDATE users UPDATE users
SET feed_needs_update = true SET feed_needs_update = true
WHERE $1 = ANY(subscriptions) WHERE $1 = ANY(subscriptions)
SQL SQL
PG_DB.exec(request, video.ucid) PG_DB.exec(request, channel_id)
end end
def update_preferences(user : User) def update_preferences(user : User)

View File

@ -1,8 +1,32 @@
struct VideoNotification
getter video_id : String
getter channel_id : String
getter published : Time
def_hash @channel_id, @video_id
def ==(other)
video_id == other.video_id
end
def self.from_video(video : ChannelVideo) : self
VideoNotification.new(video.id, video.ucid, video.published)
end
def initialize(@video_id, @channel_id, @published)
end
def clone : VideoNotification
VideoNotification.new(video_id.clone, channel_id.clone, published.clone)
end
end
class Invidious::Jobs::NotificationJob < Invidious::Jobs::BaseJob class Invidious::Jobs::NotificationJob < Invidious::Jobs::BaseJob
private getter notification_channel : ::Channel(VideoNotification)
private getter connection_channel : ::Channel({Bool, ::Channel(PQ::Notification)}) private getter connection_channel : ::Channel({Bool, ::Channel(PQ::Notification)})
private getter pg_url : URI private getter pg_url : URI
def initialize(@connection_channel, @pg_url) def initialize(@notification_channel, @connection_channel, @pg_url)
end end
def begin def begin
@ -10,6 +34,58 @@ class Invidious::Jobs::NotificationJob < Invidious::Jobs::BaseJob
PG.connect_listen(pg_url, "notifications") { |event| connections.each(&.send(event)) } PG.connect_listen(pg_url, "notifications") { |event| connections.each(&.send(event)) }
# hash of channels to their videos (id+published) that need notifying
to_notify = Hash(String, Set(VideoNotification)).new(->(hash : Hash(String, Set(VideoNotification)), key : String) { hash[key] = Set(VideoNotification).new })
# fiber to locally cache all incoming notifications (from pubsub webhooks and refresh channels job)
spawn do
begin
loop do
notification = notification_channel.receive
to_notify[notification.channel_id] << notification
end
end
end
# fiber to regularly persist all cached notifications
spawn do
loop do
begin
LOGGER.debug("NotificationJob: waking up")
cloned = to_notify.clone
to_notify.clear
cloned.each do |channel_id, notifications|
if notifications.empty?
next
end
LOGGER.info("NotificationJob: updating channel #{channel_id} with #{notifications.size} notifications")
if CONFIG.enable_user_notifications
video_ids = notifications.map { |n| n.video_id }
Invidious::Database::Users.add_multiple_notifications(channel_id, video_ids)
notifications.each do |n|
# Deliver notifications to `/api/v1/auth/notifications`
payload = {
"topic" => n.channel_id,
"videoId" => n.video_id,
"published" => n.published.to_unix,
}.to_json
PG_DB.exec("NOTIFY notifications, E'#{payload}'")
end
else
Invidious::Database::Users.feed_needs_update(channel_id)
end
end
LOGGER.trace("NotificationJob: Done, sleeping")
rescue ex
LOGGER.error("NotificationJob: #{ex.message}")
end
sleep 1.minute
Fiber.yield
end
end
loop do loop do
action, connection = connection_channel.receive action, connection = connection_channel.receive

View File

@ -425,16 +425,6 @@ module Invidious::Routes::Feeds
next # skip this video since it raised an exception (e.g. it is a scheduled live event) next # skip this video since it raised an exception (e.g. it is a scheduled live event)
end end
if CONFIG.enable_user_notifications
# Deliver notifications to `/api/v1/auth/notifications`
payload = {
"topic" => video.ucid,
"videoId" => video.id,
"published" => published.to_unix,
}.to_json
PG_DB.exec("NOTIFY notifications, E'#{payload}'")
end
video = ChannelVideo.new({ video = ChannelVideo.new({
id: id, id: id,
title: video.title, title: video.title,
@ -450,11 +440,7 @@ module Invidious::Routes::Feeds
was_insert = Invidious::Database::ChannelVideos.insert(video, with_premiere_timestamp: true) was_insert = Invidious::Database::ChannelVideos.insert(video, with_premiere_timestamp: true)
if was_insert if was_insert
if CONFIG.enable_user_notifications NOTIFICATION_CHANNEL.send(VideoNotification.from_video(video))
Invidious::Database::Users.add_notification(video)
else
Invidious::Database::Users.feed_needs_update(video)
end
end end
end end
end end