Merge 78768f18ae into 3e17d04875

.github/workflows/build-stable-container.yml

@@ -1,6 +1,7 @@
 name: Build and release container
 
 on:
+  workflow_dispatch:
   push:
     tags:
       - "v*"

CHANGELOG.md

@@ -1,6 +1,175 @@
 # CHANGELOG
 
-## 2024-04-26
+## v2.20240825.1 (2024-08-25)
+
+Add patch component to be [semver] compliant and make github actions happy.
+
+[semver]: https://semver.org/
+
+### Full list of pull requests merged since the last release (newest first)
+
+Allow manual trigger of release-container build (#4877, thanks @syeopite)
+
+
+
+## v2.20240825.0 (2024-08-25)
+
+### New features & important changes
+
+#### For users
+
+* The search bar now has a button that you can click!
+* Youtube URLs can be pasted directly in the search bar. Prepend search query with a
+  backslash (`\`) to disable that feature (useful if you need to search for a video whose
+  title contains some youtube URL).
+* On the channel page the "streams" tab can be sorted by either: "newest", "oldest" or "popular"
+* Lots of translations have been updated (thanks to our contributors on Weblate!)
+* Videos embedded in local HTML files (e.g: a webpage saved from a blog) can now be played
+
+#### For instance owners
+
+* Invidious now has the ability to provide a `po_token` and `visitordata` to Youtube in order to
+  circumvent current Youtube restrictions.
+* Invidious can use an (optional) external signature server like [inv_sig_helper]. Please note that
+  some videos can't be played without that signature server.
+* The Helm charts were moved to a separate repo: https://github.com/iv-org/invidious-helm-chart
+* We have changed how containers are released: the `latest` tag now tracks tagged releases, whereas
+  the `master` tag tracks the most recent commits of the `master` branch ("nightly" builds).
+
+[inv_sig_helper]: https://github.com/iv-org/inv_sig_helper
+
+#### For developpers
+
+* The versions of Crystal that we test in CI/CD are now: `1.9.2`, `1.10.1`, `1.11.2`, `1.12.1`.
+  Please note that due to a bug in the `libxml` bindings (See [#4256]), versions prior to `1.10.0`
+  are not recommended to use.
+* Thanks to @syeopite, the code is now [ameba] compliant.
+* Ameba is part of our CI/CD pipeline, and its rules will be enforced in future PRs.
+* The transcript code has been rewritten to permit transcripts as a feature rather than being
+  only a workaround for captions. Trancripts feature is coming soon!
+* Various fixes regarding the logic interacting with Youtube
+* The `sort_by` parameter can be used on the `/api/v1/channels/{id}/streams` endpoint. Accepted
+  values are: "newest", "oldest" and "popular"
+
+[ameba]: https://github.com/crystal-ameba/ameba
+[#4256]: https://github.com/iv-org/invidious/issues/4256
+
+
+### Bugs fixed
+
+#### User-side
+
+* Channels: fixed broken "subscribers" and "views" counters
+* Watch page: playback position is reset at the end of a video, so that the next time this video
+  is watched, it will start from the beginning rather than 15 seconds before the end
+* Watch page: the items in the "add to playlist" drop down are now sorted alphabetically
+* Videos: the "genre" URL is now always pointing to a valid webpage
+* Playlists: Fixed `Could not parse N episodes` error on podcast playlists
+* All external links should now have the [`rel`] attibute set to `noreferrer noopener` for
+  increased privacy.
+* Preferences: Fixed the admin-only "modified source code" input being ignored
+* Watch/channel pages: use the full image URL in `og:image` and `twitter:image` meta tags
+
+[`rel`]: https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel
+
+#### API
+
+* fixed the `local` parameter not applying to `formatStreams` on `/api/v1/videos/{id}`
+* fixed an `Index out of bounds` error hapenning when a playlist had no videos
+* fixed duplicated query parameters in proxied video URLs
+* Return actual video height/width/fps rather than hard coded values
+* Fixed the `/api/v1/popular` endpoint not returning a proper error code/message when the
+  popular page/endpoint are disabled.
+
+
+### Full list of pull requests merged since the last release (newest first)
+
+* HTML: Sort playlists alphabetically in watch page drop down ([#4853], by @SamantazFox)
+* Videos: Fix XSS vulnerability in description/comments ([#4852], thanks _anonymous_)
+* YtAPI: Bump client versions ([#4849], by @SamantazFox)
+* SigHelper: Fix inverted time comparison in 'check_update' ([#4845], by @SamantazFox)
+* Storyboards: Various fixes and code cleaning ([#4153], by SamantazFox)
+* Fix lint errors introduced in #4146 and #4295 ([#4876], thanks @syeopite)
+* Search: Add support for Youtube URLs ([#4146], by @SamantazFox)
+* Channel: Render age restricted channels ([#4295], thanks @ChunkyProgrammer)
+* Ameba: Miscellaneous fixes ([#4807], thanks @syeopite)
+* API: Proxy formatStreams URLs too ([#4859], thanks @colinleroy)
+* UI: Add search button to search bar ([#4706], thanks @thansk)
+* Add ability to set po_token and visitordata ID ([#4789], thanks @unixfox)
+* Add support for an external signature server ([#4772], by @SamantazFox)
+* Ameba: Fix Naming/VariableNames ([#4790], thanks @syeopite)
+* Translations update from Hosted Weblate ([#4659])
+* Ameba: Fix Lint/UselessAssign ([#4795], thanks @syeopite)
+* HTML: Add rel="noreferrer noopener" to external links ([#4667], thanks @ulmemxpoc)
+* Remove unused methods in Invidious::LogHandler ([#4812], thanks @syeopite)
+* Ameba: Fix Lint/NotNilAfterNoBang ([#4796], thanks @syeopite)
+* Ameba: Fix unused argument Lint warnings ([#4805], thanks @syeopite)
+* Ameba: i18next.cr fixes ([#4806], thanks @syeopite)
+* Ameba: Disable rules ([#4792], thanks @syeopite)
+* Channel: parse subscriber count and channel banner ([#4785], thanks @ChunkyProgrammer)
+* Player: Fix playback position of already watched videos ([#4731], thanks @Fijxu)
+* Videos: Fix genre url being unusable ([#4717], thanks @meatball133)
+* API: Fix out of bound error on empty playlists ([#4696], thanks @Fijxu)
+* Handle playlists cataloged as Podcast ([#4695], thanks @Fijxu)
+* API: Fix duplicated query parameters in proxied video URLs ([#4587], thanks @absidue)
+* API: Return actual stream height, width and fps ([#4586], thanks @absidue)
+* Preferences: Fix handling of modified source code URL ([#4437], thanks @nooptek)
+* API: Fix URL for vtt subtitles ([#4221], thanks @karelrooted)
+* Channels: Add sort options to streams ([#4224], thanks @src-tinkerer)
+* API: Fix error code for disabled popular endpoint ([#4296], thanks @iBicha)
+* Allow embedding videos in local HTML files ([#4450], thanks @tomasz1986)
+* CI: Bump Crystal version matrix ([#4654], by @SamantazFox)
+* YtAPI: Remove API keys like official clients ([#4655], by @SamantazFox)
+* HTML: Use full URL in the og:image property ([#4675], thanks @Fijxu)
+* Rewrite transcript logic to be more generic ([#4747], thanks @syeopite)
+* CI: Run Ameba ([#4753], thanks @syeopite)
+* CI: Add release based containers ([#4763], thanks @syeopite)
+* move helm chart to a dedicated github repository ([#4711], thanks @unixfox)
+
+[#4146]: https://github.com/iv-org/invidious/pull/4146
+[#4153]: https://github.com/iv-org/invidious/pull/4153
+[#4221]: https://github.com/iv-org/invidious/pull/4221
+[#4224]: https://github.com/iv-org/invidious/pull/4224
+[#4295]: https://github.com/iv-org/invidious/pull/4295
+[#4296]: https://github.com/iv-org/invidious/pull/4296
+[#4437]: https://github.com/iv-org/invidious/pull/4437
+[#4450]: https://github.com/iv-org/invidious/pull/4450
+[#4586]: https://github.com/iv-org/invidious/pull/4586
+[#4587]: https://github.com/iv-org/invidious/pull/4587
+[#4654]: https://github.com/iv-org/invidious/pull/4654
+[#4655]: https://github.com/iv-org/invidious/pull/4655
+[#4659]: https://github.com/iv-org/invidious/pull/4659
+[#4667]: https://github.com/iv-org/invidious/pull/4667
+[#4675]: https://github.com/iv-org/invidious/pull/4675
+[#4695]: https://github.com/iv-org/invidious/pull/4695
+[#4696]: https://github.com/iv-org/invidious/pull/4696
+[#4706]: https://github.com/iv-org/invidious/pull/4706
+[#4711]: https://github.com/iv-org/invidious/pull/4711
+[#4717]: https://github.com/iv-org/invidious/pull/4717
+[#4731]: https://github.com/iv-org/invidious/pull/4731
+[#4747]: https://github.com/iv-org/invidious/pull/4747
+[#4753]: https://github.com/iv-org/invidious/pull/4753
+[#4763]: https://github.com/iv-org/invidious/pull/4763
+[#4772]: https://github.com/iv-org/invidious/pull/4772
+[#4785]: https://github.com/iv-org/invidious/pull/4785
+[#4789]: https://github.com/iv-org/invidious/pull/4789
+[#4790]: https://github.com/iv-org/invidious/pull/4790
+[#4792]: https://github.com/iv-org/invidious/pull/4792
+[#4795]: https://github.com/iv-org/invidious/pull/4795
+[#4796]: https://github.com/iv-org/invidious/pull/4796
+[#4805]: https://github.com/iv-org/invidious/pull/4805
+[#4806]: https://github.com/iv-org/invidious/pull/4806
+[#4807]: https://github.com/iv-org/invidious/pull/4807
+[#4812]: https://github.com/iv-org/invidious/pull/4812
+[#4845]: https://github.com/iv-org/invidious/pull/4845
+[#4849]: https://github.com/iv-org/invidious/pull/4849
+[#4852]: https://github.com/iv-org/invidious/pull/4852
+[#4853]: https://github.com/iv-org/invidious/pull/4853
+[#4859]: https://github.com/iv-org/invidious/pull/4859
+[#4876]: https://github.com/iv-org/invidious/pull/4876
+
+
+## v2.20240427 (2024-04-27)
 
 Major bug fixes:
  * Videos: Use android test suite client (#4650, thanks @SamantazFox)

src/invidious.cr

@@ -184,8 +184,9 @@ if CONFIG.popular_enabled
   Invidious::Jobs.register Invidious::Jobs::PullPopularVideosJob.new(PG_DB)
 end
 
-CONNECTION_CHANNEL = ::Channel({Bool, ::Channel(PQ::Notification)}).new(32)
+NOTIFICATION_CHANNEL = ::Channel(VideoNotification).new(32)
-Invidious::Jobs.register Invidious::Jobs::NotificationJob.new(CONNECTION_CHANNEL, CONFIG.database_url)
+CONNECTION_CHANNEL   = ::Channel({Bool, ::Channel(PQ::Notification)}).new(32)
+Invidious::Jobs.register Invidious::Jobs::NotificationJob.new(NOTIFICATION_CHANNEL, CONNECTION_CHANNEL, CONFIG.database_url)
 
 Invidious::Jobs.register Invidious::Jobs::ClearExpiredItemsJob.new
 

src/invidious/channels/channels.cr

@@ -249,11 +249,7 @@ def fetch_channel(ucid, pull_all_videos : Bool)
 
     if was_insert
       LOGGER.trace("fetch_channel: #{ucid} : video #{video_id} : Inserted, updating subscriptions")
-      if CONFIG.enable_user_notifications
+      NOTIFICATION_CHANNEL.send(VideoNotification.from_video(video))
-        Invidious::Database::Users.add_notification(video)
-      else
-        Invidious::Database::Users.feed_needs_update(video)
-      end
     else
       LOGGER.trace("fetch_channel: #{ucid} : video #{video_id} : Updated")
     end
@@ -285,11 +281,7 @@ def fetch_channel(ucid, pull_all_videos : Bool)
         if Time.utc - video.published > 1.minute
           was_insert = Invidious::Database::ChannelVideos.insert(video)
           if was_insert
-            if CONFIG.enable_user_notifications
+            NOTIFICATION_CHANNEL.send(VideoNotification.from_video(video))
-              Invidious::Database::Users.add_notification(video)
-            else
-              Invidious::Database::Users.feed_needs_update(video)
-            end
           end
         end
       end

src/invidious/database/users.cr

@@ -119,15 +119,15 @@ module Invidious::Database::Users
   #  Update (notifs)
   # -------------------
 
-  def add_notification(video : ChannelVideo)
+  def add_multiple_notifications(channel_id : String, video_ids : Array(String))
     request = <<-SQL
       UPDATE users
-      SET notifications = array_append(notifications, $1),
+      SET notifications = array_cat(notifications, $1),
           feed_needs_update = true
       WHERE $2 = ANY(subscriptions)
     SQL
 
-    PG_DB.exec(request, video.id, video.ucid)
+    PG_DB.exec(request, video_ids, channel_id)
   end
 
   def remove_notification(user : User, vid : String)
@@ -154,14 +154,14 @@ module Invidious::Database::Users
   #  Update (misc)
   # -------------------
 
-  def feed_needs_update(video : ChannelVideo)
+  def feed_needs_update(channel_id : String)
     request = <<-SQL
       UPDATE users
       SET feed_needs_update = true
       WHERE $1 = ANY(subscriptions)
     SQL
 
-    PG_DB.exec(request, video.ucid)
+    PG_DB.exec(request, channel_id)
   end
 
   def update_preferences(user : User)

src/invidious/jobs/notification_job.cr

@@ -1,8 +1,32 @@
+struct VideoNotification
+  getter video_id : String
+  getter channel_id : String
+  getter published : Time
+
+  def_hash @channel_id, @video_id
+
+  def ==(other)
+    video_id == other.video_id
+  end
+
+  def self.from_video(video : ChannelVideo) : self
+    VideoNotification.new(video.id, video.ucid, video.published)
+  end
+
+  def initialize(@video_id, @channel_id, @published)
+  end
+
+  def clone : VideoNotification
+    VideoNotification.new(video_id.clone, channel_id.clone, published.clone)
+  end
+end
+
 class Invidious::Jobs::NotificationJob < Invidious::Jobs::BaseJob
+  private getter notification_channel : ::Channel(VideoNotification)
   private getter connection_channel : ::Channel({Bool, ::Channel(PQ::Notification)})
   private getter pg_url : URI
 
-  def initialize(@connection_channel, @pg_url)
+  def initialize(@notification_channel, @connection_channel, @pg_url)
   end
 
   def begin
@@ -10,6 +34,58 @@ class Invidious::Jobs::NotificationJob < Invidious::Jobs::BaseJob
 
     PG.connect_listen(pg_url, "notifications") { |event| connections.each(&.send(event)) }
 
+    # hash of channels to their videos (id+published) that need notifying
+    to_notify = Hash(String, Set(VideoNotification)).new(->(hash : Hash(String, Set(VideoNotification)), key : String) { hash[key] = Set(VideoNotification).new })
+
+    # fiber to locally cache all incoming notifications (from pubsub webhooks and refresh channels job)
+    spawn do
+      begin
+        loop do
+          notification = notification_channel.receive
+          to_notify[notification.channel_id] << notification
+        end
+      end
+    end
+    # fiber to regularly persist all cached notifications
+    spawn do
+      loop do
+        begin
+          LOGGER.debug("NotificationJob: waking up")
+          cloned = to_notify.clone
+          to_notify.clear
+
+          cloned.each do |channel_id, notifications|
+            if notifications.empty?
+              next
+            end
+
+            LOGGER.info("NotificationJob: updating channel #{channel_id} with #{notifications.size} notifications")
+            if CONFIG.enable_user_notifications
+              video_ids = notifications.map { |n| n.video_id }
+              Invidious::Database::Users.add_multiple_notifications(channel_id, video_ids)
+              notifications.each do |n|
+                # Deliver notifications to `/api/v1/auth/notifications`
+                payload = {
+                  "topic"     => n.channel_id,
+                  "videoId"   => n.video_id,
+                  "published" => n.published.to_unix,
+                }.to_json
+                PG_DB.exec("NOTIFY notifications, E'#{payload}'")
+              end
+            else
+              Invidious::Database::Users.feed_needs_update(channel_id)
+            end
+          end
+
+          LOGGER.trace("NotificationJob: Done, sleeping")
+        rescue ex
+          LOGGER.error("NotificationJob: #{ex.message}")
+        end
+        sleep 1.minute
+        Fiber.yield
+      end
+    end
+
     loop do
       action, connection = connection_channel.receive
 

src/invidious/routes/feeds.cr

@@ -425,16 +425,6 @@ module Invidious::Routes::Feeds
           next # skip this video since it raised an exception (e.g. it is a scheduled live event)
         end
 
-        if CONFIG.enable_user_notifications
-          # Deliver notifications to `/api/v1/auth/notifications`
-          payload = {
-            "topic"     => video.ucid,
-            "videoId"   => video.id,
-            "published" => published.to_unix,
-          }.to_json
-          PG_DB.exec("NOTIFY notifications, E'#{payload}'")
-        end
-
         video = ChannelVideo.new({
           id:                 id,
           title:              video.title,
@@ -450,11 +440,7 @@ module Invidious::Routes::Feeds
 
         was_insert = Invidious::Database::ChannelVideos.insert(video, with_premiere_timestamp: true)
         if was_insert
-          if CONFIG.enable_user_notifications
+          NOTIFICATION_CHANNEL.send(VideoNotification.from_video(video))
-            Invidious::Database::Users.add_notification(video)
-          else
-            Invidious::Database::Users.feed_needs_update(video)
-          end
         end
       end
     end