limit feeds and delete materialized views

CHANGELOG.md

@@ -6,6 +6,79 @@
 
 
 
+
+## v2.20261103.0
+
+### Wrap-up
+
+This release brings several user-facing improvements: custom channel handle URLs with tab paths, a minute badge on the watch page, automatic video start when clicking timestamps, support for the shows tab on channels, and a new subscription export for NewPipe. It also fixes parsing of published dates and view counts for collab videos and corrects the OPML export for FreeTube. On the breaking side, the hd720 quality option has been removed and Reddit integration has been completely dropped.
+
+For instance owners, Invidious now checks domain configuration before starting, allows adding custom text at the bottom of pages, and improves companion proxying. The toolchain has been updated to Crystal 1.22.2, and Nix packaging has been refreshed with multiple nixpkgs bumps.
+
+Developers will find improved spec handling with a `-v` flag, removal of unused functions, better module encapsulation, and fixes for deprecation warnings across Crystal versions. CI dependencies were also updated.
+
+### New features & important changes
+
+#### For Users
+- Support for custom channel handle URLs with tab paths, e.g. `{INSTANCE}/markrober/streams` (#5811)
+- Added a minute badge next to the video title on the watch page (#6112)
+- Clicking a timestamp now automatically starts the video and scrolls to it (#6111)
+- Added support for the shows tab on channels (#6033)
+- New subscription export in JSON format for NewPipe (#5826)
+- Removed the hd720 quality option (#6098)
+- Removed all Reddit integration (#6087)
+
+#### For instance owners
+- Invidious now checks domain configuration before starting (#5352)
+- Added ability to display custom text at the bottom of pages, e.g. for contacting a maintainer (#3826)
+- Improved companion proxying (#5809)
+- Updated to Crystal 1.22.2 (#6106, #6099)
+- Nix packaging bumps: nixpkgs and flake-utils updates (#6102, #6086, #6097)
+
+#### For developers
+- Specs now support a `-v` flag for verbose output (#6115)
+- Removed unused functions `.ci_lower_bound()` and `.elapsed_text()` (#6110)
+- Moved FreeTube OPML export to the Export module (#6107)
+- Encapsulated more functions into modules and improved spec files (#6046)
+- Fixed deprecation warnings across Crystal versions (#6100)
+- Updated CI dependency `crystal-lang/install-crystal` (#6096)
+
+### Bugs fixed
+#### For Users
+- Fixed parsing of published date and view count for collab videos (#5812)
+- Fixed OPML export for FreeTube (#6109)
+
+#### For developers
+- Fixed deprecation warnings that appeared across different Crystal versions (#6100)
+
+### Full list of pull requests merged since the last release (newest first)
+
+* fix(parser): parse published date and view count for collab videos (https://github.com/iv-org/invidious/pull/5812, by @radmorecameron)
+* feat: Adds support for custom channel handle urls with tab paths (ex: `{INSTANCE}/markrober/streams`) (https://github.com/iv-org/invidious/pull/5811, by @radmorecameron)
+* chore: improve invidious companion proxying (https://github.com/iv-org/invidious/pull/5809, by @Fijxu)
+* check domain configuration before starting Invidious (https://github.com/iv-org/invidious/pull/5352, by @Fijxu)
+* chore: use `-v` cmdline argument for verbose output of the specs (https://github.com/iv-org/invidious/pull/6115, by @Fijxu)
+* feat: add minute badge to the watch page, right next to the title of the video (https://github.com/iv-org/invidious/pull/6112, by @Fijxu)
+* Add custom text at the bottom - contact a maintainer (https://github.com/iv-org/invidious/pull/3826, by @auralglow)
+* feat: automatically start video and scroll to it if clicking timestam… (https://github.com/iv-org/invidious/pull/6111, by @Fijxu)
+* chore: remove unused functions .ci_lower_bound() and .elapsed_text() (https://github.com/iv-org/invidious/pull/6110, by @Fijxu)
+* fix: fix the OPML export for FreeTube (https://github.com/iv-org/invidious/pull/6109, by @Fijxu)
+* chore: move FreeTube OPML export to Export module (https://github.com/iv-org/invidious/pull/6107, by @Fijxu)
+* Translations update from Hosted Weblate (https://github.com/iv-org/invidious/pull/5949, by @weblate)
+* chore: update to Crystal 1.22.2 (https://github.com/iv-org/invidious/pull/6106, by @Fijxu)
+* Add subscription JSON export for NewPipe (https://github.com/iv-org/invidious/pull/5826, by @unrealtournament)
+* chore: Encapsulate more functions into their own modules and improve spec files (https://github.com/iv-org/invidious/pull/6046, by @Fijxu)
+* chore(deps): bump nixpkgs from `02c087f` to `3ec6363` in /nix (https://github.com/iv-org/invidious/pull/6102, by @dependabot[bot])
+* chore(deps): bump flake-utils from `11707dc` to `b7a79d3` in /nix (https://github.com/iv-org/invidious/pull/6086, by @dependabot[bot])
+* chore: remove hd720 quality (https://github.com/iv-org/invidious/pull/6098, by @Fijxu)
+* feat(channel): add support for shows tab (https://github.com/iv-org/invidious/pull/6033, by @radmorecameron)
+* fix: fix warnings from deprecations across Crystal versions (https://github.com/iv-org/invidious/pull/6100, by @Fijxu)
+* chore: Completely remove Reddit from Invidious (https://github.com/iv-org/invidious/pull/6087, by @Fijxu)
+* chore: update to Crystal 1.22.1 (https://github.com/iv-org/invidious/pull/6099, by @Fijxu)
+* chore(deps): bump nixpkgs from `151fa4e` to `02c087f` in /nix (https://github.com/iv-org/invidious/pull/6097, by @dependabot[bot])
+* chore(deps): bump crystal-lang/install-crystal from 1.9.3 to 1.9.4 (https://github.com/iv-org/invidious/pull/6096, by @dependabot[bot])
+* Release v2.20261022.0 (https://github.com/iv-org/invidious/pull/6093, by @github-actions[bot])
+
 ## v2.20261022.0
 
 ### Wrap-up

config/migrate-scripts/migrate-db-8bc91ce.sh

@@ -0,0 +1,6 @@
+CREATE INDEX channel_videos_ucid_published_idx
+  ON public.channel_videos
+  USING btree
+  (ucid COLLATE pg_catalog."default", published);
+
+DROP INDEX channel_videos_ucid_idx;

config/sql/channel_videos.sql

@@ -19,12 +19,12 @@ CREATE TABLE IF NOT EXISTS public.channel_videos
 
 GRANT ALL ON TABLE public.channel_videos TO current_user;
 
--- Index: public.channel_videos_ucid_idx
+-- Index: public.channel_videos_ucid_published_idx
 
--- DROP INDEX public.channel_videos_ucid_idx;
+-- DROP INDEX public.channel_videos_ucid_published_idx;
 
-CREATE INDEX IF NOT EXISTS channel_videos_ucid_idx
+CREATE INDEX IF NOT EXISTS channel_videos_ucid_published_idx
   ON public.channel_videos
   USING btree
-  (ucid COLLATE pg_catalog."default");
+  (ucid COLLATE pg_catalog."default", published);
 

shard.yml

@@ -1,5 +1,5 @@
 name: invidious
-version: 2.20261022.0-dev
+version: 2.20261103.0-dev
 
 authors:
   - Invidious team <contact@invidious.io>

src/invidious.cr

@@ -130,14 +130,6 @@ Kemal.config.extra_options do |parser|
       exit
     end
   end
-  parser.on("-f THREADS", "--feed-threads=THREADS", "Number of threads for refreshing feeds (default: #{CONFIG.feed_threads})") do |number|
-    begin
-      CONFIG.feed_threads = number.to_i
-    rescue
-      puts "THREADS must be integer"
-      exit
-    end
-  end
   parser.on("-o OUTPUT", "--output=OUTPUT", "Redirect output (default: #{CONFIG.output})") do |output|
     CONFIG.output = output
   end
@@ -189,10 +181,6 @@ if CONFIG.channel_threads > 0
   Invidious::Jobs.register Invidious::Jobs::RefreshChannelsJob.new(PG_DB)
 end
 
-if CONFIG.feed_threads > 0
-  Invidious::Jobs.register Invidious::Jobs::RefreshFeedsJob.new(PG_DB)
-end
-
 if CONFIG.statistics_enabled
   Invidious::Jobs.register Invidious::Jobs::StatisticsRefreshJob.new(PG_DB, SOFTWARE)
 end

src/invidious/config.cr

@@ -101,8 +101,6 @@ class Config
   # Time interval between two executions of the job that crawls channel videos (subscriptions update).
   @[YAML::Field(converter: Preferences::TimeSpanConverter)]
   property channel_refresh_interval : Time::Span = 30.minutes
-  # Number of threads to use for updating feeds
-  property feed_threads : Int32 = 1
   # Log file path or STDOUT
   property output : String = "STDOUT"
   # Default log level, valid YAML values are ints and strings, see src/invidious/helpers/logger.cr

src/invidious/jobs/refresh_feeds_job.cr

@@ -1,75 +0,0 @@
-class Invidious::Jobs::RefreshFeedsJob < Invidious::Jobs::BaseJob
-  private getter db : DB::Database
-
-  def initialize(@db)
-  end
-
-  def begin
-    max_fibers = CONFIG.feed_threads
-    active_fibers = 0
-    active_channel = ::Channel(Bool).new
-
-    loop do
-      db.query("SELECT email FROM users WHERE feed_needs_update = true OR feed_needs_update IS NULL") do |rs|
-        rs.each do
-          email = rs.read(String)
-          view_name = "subscriptions_#{sha256(email)}"
-
-          if active_fibers >= max_fibers
-            if active_channel.receive
-              active_fibers -= 1
-            end
-          end
-
-          active_fibers += 1
-          spawn do
-            begin
-              # Drop outdated views
-              column_array = Invidious::Database.get_column_array(db, view_name)
-              ChannelVideo.type_array.each_with_index do |name, i|
-                if name != column_array[i]?
-                  LOGGER.info("RefreshFeedsJob: DROP MATERIALIZED VIEW #{view_name}")
-                  db.exec("DROP MATERIALIZED VIEW #{view_name}")
-                  raise "view does not exist"
-                end
-              end
-
-              if !db.query_one("SELECT pg_get_viewdef('#{view_name}')", as: String).includes? "WHERE ((cv.ucid = ANY (u.subscriptions))"
-                LOGGER.info("RefreshFeedsJob: Materialized view #{view_name} is out-of-date, recreating...")
-                db.exec("DROP MATERIALIZED VIEW #{view_name}")
-              end
-
-              db.exec("REFRESH MATERIALIZED VIEW #{view_name}")
-              db.exec("UPDATE users SET feed_needs_update = false WHERE email = $1", email)
-            rescue ex
-              # Rename old views
-              begin
-                legacy_view_name = "subscriptions_#{sha256(email)[0..7]}"
-
-                db.exec("SELECT * FROM #{legacy_view_name} LIMIT 0")
-                LOGGER.info("RefreshFeedsJob: RENAME MATERIALIZED VIEW #{legacy_view_name}")
-                db.exec("ALTER MATERIALIZED VIEW #{legacy_view_name} RENAME TO #{view_name}")
-              rescue ex
-                begin
-                  # While iterating through, we may have an email stored from a deleted account
-                  if db.query_one?("SELECT true FROM users WHERE email = $1", email, as: Bool)
-                    LOGGER.info("RefreshFeedsJob: CREATE #{view_name}")
-                    db.exec("CREATE MATERIALIZED VIEW #{view_name} AS #{Invidious::User::Users::MATERIALIZED_VIEW_SQL.call(email)}")
-                    db.exec("UPDATE users SET feed_needs_update = false WHERE email = $1", email)
-                  end
-                rescue ex
-                  LOGGER.error("RefreshFeedJobs: REFRESH #{email} : #{ex.message}")
-                end
-              end
-            end
-
-            active_channel.send(true)
-          end
-        end
-      end
-
-      sleep 5.seconds
-      Fiber.yield
-    end
-  end
-end

src/invidious/routes/account.cr

@@ -123,10 +123,8 @@ module Invidious::Routes::Account
       return Errors.error_template(400, ex)
     end
 
-    view_name = "subscriptions_#{sha256(user.email)}"
     Invidious::Database::Users.delete(user)
     Invidious::Database::SessionIDs.delete(email: user.email)
-    PG_DB.exec("DROP MATERIALIZED VIEW #{view_name}")
 
     env.request.cookies.each do |cookie|
       cookie.expires = Time.utc(1990, 1, 1)

src/invidious/routes/login.cr

@@ -125,9 +125,6 @@ module Invidious::Routes::Login
         Invidious::Database::Users.insert(user)
         Invidious::Database::SessionIDs.insert(sid, email)
 
-        view_name = "subscriptions_#{sha256(user.email)}"
-        PG_DB.exec("CREATE MATERIALIZED VIEW #{view_name} AS #{Invidious::User::Users::MATERIALIZED_VIEW_SQL.call(user.email)}")
-
         if alt = CONFIG.alternative_domains.index(host)
           env.response.cookies["SID"] = Invidious::User::Cookies.sid(CONFIG.alternative_domains[alt], sid)
         else

src/invidious/search/processors.cr

@@ -36,19 +36,19 @@ module Invidious::Search::Processors
 
   # Search inside of user subscriptions
   def subscriptions(query : Query, user : Invidious::User) : Array(ChannelVideo)
-    view_name = "subscriptions_#{sha256(user.email)}"
-
     return PG_DB.query_all(<<-SQL,
       SELECT id,title,published,updated,ucid,author,length_seconds
       FROM (
-        SELECT *,
-        to_tsvector(#{view_name}.title) ||
-        to_tsvector(#{view_name}.author)
-        as document
-        FROM #{view_name}
-      ) v_search WHERE v_search.document @@ plainto_tsquery($1) LIMIT 20 OFFSET $2;
+        SELECT cv.*,
+        to_tsvector(cv.title) ||
+        to_tsvector(cv.author) AS document
+        FROM channel_videos cv
+        JOIN users ON cv.ucid = any(users.subscriptions)
+        WHERE users.email = $1 AND published > now() - interval '1 month'
+        ORDER BY published
+      ) v_search WHERE v_search.document @@ plainto_tsquery($2) LIMIT 20 OFFSET $3;
       SQL
-      query.text, (query.page - 1) * 20,
+      user.email, query.text, (query.page - 1) * 20,
       as: ChannelVideo
     )
   end

src/invidious/user/users.cr

@@ -2,8 +2,6 @@ require "crypto/bcrypt/password"
 
 module Invidious::User::Users
   extend self
-  # Materialized views may not be defined using bound parameters (`$1` as used elsewhere)
-  MATERIALIZED_VIEW_SQL = ->(email : String) { "SELECT cv.* FROM channel_videos cv WHERE EXISTS (SELECT subscriptions FROM users u WHERE cv.ucid = ANY (u.subscriptions) AND u.email = E'#{email.gsub({'\'' => "\\'", '\\' => "\\\\"})}') ORDER BY published DESC" }
 
   def create_user(sid, email, password)
     password = Crypto::Bcrypt::Password.create(password, cost: 10)
@@ -29,7 +27,6 @@ module Invidious::User::Users
     offset = (page - 1) * limit
 
     notifications = Invidious::Database::Users.select_notifications(user)
-    view_name = "subscriptions_#{sha256(user.email)}"
 
     if user.preferences.notifications_only && !notifications.empty?
       # Only show notifications
@@ -54,11 +51,19 @@ module Invidious::User::Users
           # Show latest video from a channel that a user hasn't watched
           # "unseen_only" isn't really correct here, more accurate would be "unwatched_only"
 
-          videos = PG_DB.query_all("SELECT DISTINCT ON (ucid) * FROM #{view_name} WHERE NOT id = ANY ($1::text[]) ORDER BY ucid, published DESC", user.watched, as: ChannelVideo)
+          videos = PG_DB.query_all("SELECT DISTINCT ON (cv.ucid) cv.* " \
+                                   "FROM channel_videos cv " \
+                                   "JOIN users ON cv.ucid = any(users.subscriptions) " \
+                                   "WHERE users.email = $1 AND NOT cv.id = any(users.watched) AND published > now() - interval '1 month' " \
+                                   "ORDER BY ucid, published DESC", user.email, as: ChannelVideo)
         else
           # Show latest video from each channel
 
-          videos = PG_DB.query_all("SELECT DISTINCT ON (ucid) * FROM #{view_name} ORDER BY ucid, published DESC", as: ChannelVideo)
+          videos = PG_DB.query_all("SELECT DISTINCT ON (cv.ucid) cv.* " \
+                                   "FROM channel_videos cv " \
+                                   "JOIN users ON cv.ucid = any(users.subscriptions) " \
+                                   "WHERE users.email = $1 AND published > now() - interval '1 month' " \
+                                   "ORDER BY ucid, published DESC", user.email, as: ChannelVideo)
         end
 
         videos.sort_by!(&.published).reverse!
@@ -66,11 +71,19 @@ module Invidious::User::Users
         if user.preferences.unseen_only
           # Only show unwatched
 
-          videos = PG_DB.query_all("SELECT * FROM #{view_name} WHERE NOT id = ANY ($1::text[]) ORDER BY published DESC LIMIT $2 OFFSET $3", user.watched, limit, offset, as: ChannelVideo)
+          videos = PG_DB.query_all("SELECT cv.* " \
+                                   "FROM channel_videos cv " \
+                                   "JOIN users ON cv.ucid = any(users.subscriptions) " \
+                                   "WHERE users.email = $1 AND NOT cv.id = any(users.watched) AND published > now() - interval '1 month' " \
+                                   "ORDER BY published DESC LIMIT $2 OFFSET $3", user.email, limit, offset, as: ChannelVideo)
         else
           # Sort subscriptions as normal
 
-          videos = PG_DB.query_all("SELECT * FROM #{view_name} ORDER BY published DESC LIMIT $1 OFFSET $2", limit, offset, as: ChannelVideo)
+          videos = PG_DB.query_all("SELECT cv.* " \
+                                   "FROM channel_videos cv " \
+                                   "JOIN users ON cv.ucid = any(users.subscriptions) " \
+                                   "WHERE users.email = $1 AND published > now() - interval '1 month' " \
+                                   "ORDER BY published DESC LIMIT $2 OFFSET $3", user.email, limit, offset, as: ChannelVideo)
         end
       end