Handling file uploads

I’m trying to add file uploads to my project and I’m a bit lost. I’ve gotten as far as receiving a Tempfile in my action, but I’m wary of doing too much manual work if there is a better way. I’ve seen Shrine suggested, but it doesn’t have any Hanakai-based examples on its page and any resources around using it are outdated since the release of Hanami 3.

there was also this thread about a canonical file storage solution and the linked article was a good start, but a lot of the code examples don’t seem to work on a vanilla Hanami installation or requires a lot of existing knowledge of how the systems work which I don’t have :confused:

I guess the question is: as a Ruby noobie with a fresh Hanami project, how do I best add file uploads?

I just set up a brand new Hanami 3 app with Shrine last month. It’s still a work in progress though, and there may be bugs or edge cases. I’ll polish it over the next weeks/months, but it works well.

Here’s the provider to set up and register Shrine:

# config/providers/shrine.rb

# frozen_string_literal: true

Hanami.app.register_provider :shrine do
  prepare do
    require "shrine"
    require "shrine/storage/file_system"
    require "shrine/storage/memory"
  end

  start do
    if Hanami.env?(:test)
      Shrine.storages = {
        cache: Shrine::Storage::Memory.new,
        store: Shrine::Storage::Memory.new
      }
    else
      Shrine.storages = {
        cache: Shrine::Storage::FileSystem.new(".", prefix: "uploads/cache"),
        store: Shrine::Storage::FileSystem.new(".", prefix: "uploads/store")
      }
    end
    
    Shrine.plugin :determine_mime_type
    Shrine.plugin :validation_helpers
    Shrine.plugin :pretty_location
    Shrine.plugin :derivatives
    Shrine.plugin MyApp::Shrine::Plugins::Attach
  end
end

The uploader is fairly simple:

# auto_register: false
# frozen_string_literal: true

module MyApp
  module Uploaders
    class AvatarUploader < ::Shrine
      MAX_MB = 5
      MAX_BYTES = MAX_MB * 1024 * 1024

      def self.message_args
        {max_mb: MAX_MB}
      end

      Attacher.validate do
        validate_max_size MAX_BYTES, message: "too_large"
        validate_mime_type %w[image/jpeg image/png image/webp image/avif],
          message: "unsupported_type"
      end
    end
  end
end

The glue for Hanami comes in two parts. The first is a plugin for Shrine as registered above:

# lib/shrine/plugins/attach.rb

# auto_register: false
# frozen_string_literal: true

require "json"

module MyApp
  module Shrine
    module Plugins
      module Attach
        module ClassMethods
          # Returns [cache_data, errors], data is nil when absent or invalid.
          def cache(upload)
            attacher = self::Attacher.new
            if (io = extract_io(upload))
              attacher.attach_cached(io)
            end
            return [nil, attacher.errors] if attacher.errors.any?
            return [nil, []] unless attacher.file

            [attacher.data, []]
          end

          # Reuses cached file data from a hidden field after a re-render.
          def reuse(json)
            return unless json.is_a?(String) && !json.empty?

            JSON.parse(json)
          rescue JSON::ParserError
            nil
          end

          # Moves cache to :store and deletes the cache copy and data already
          # in :store is returned untouched.
          def promote(data)
            return unless data

            cached = uploaded_file(data)
            return data unless self::Attacher.new.cached?(cached)

            attacher = self::Attacher.new
            attacher.assign(cached)
            attacher.promote_cached
            cached.delete
            attacher.data
          end

          def detach(data)
            return unless data

            uploaded_file(data).delete
          end

          def url_for(data)
            return unless data

            uploaded_file(data).url
          end

          private

          def extract_io(upload)
            return upload unless upload.is_a?(Hash)

            io = upload[:tempfile]
            io if io.respond_to?(:size) && io.size.to_i.positive?
          end
        end
      end
    end
  end
end

The second part is a mixin for operations adding convenience methods to handle attaching/promotion/detaching etc.

# lib/operations/attachment.rb

# auto_register: false
# frozen_string_literal: true

require "json"

module MyApp
  module Operations
    module Attachment
      def self.included(base)
        base.extend(ClassMethods)
      end

      module ClassMethods
        def attach(name, using:)
          define_attach_helpers(name, using)
          define_storage_helpers(name, using)

          private :"attach_#{name}",
            :"cached_#{name}_json",
            :"promote_#{name}",
            :"merge_#{name}",
            :"detach_#{name}"
        end

        private

        def define_attach_helpers(name, uploader)
          error_base = "#{self.name.split("::").first.downcase}.errors.#{name}"

          define_method("attach_#{name}") do |upload: nil, cached_json: nil, remove: false|
            data, errors = uploader.cache(upload)
            next Failure(name => attachment_errors(error_base, errors, uploader)) if errors.any?
            next Success(false) if data.nil? && flagged(remove)

            data ||= uploader.reuse(cached_json)
            Success(data)
          end

          define_method("cached_#{name}_json") do |data|
            data ? JSON.generate(data) : nil
          end
        end

        def define_storage_helpers(name, uploader)
          column = :"#{name}_data"

          define_method("promote_#{name}") do |data|
            uploader.promote(data)
          end

          # nil leaves the column alone, false clears it, data sets it.
          define_method("merge_#{name}") do |attrs, data|
            case data
            when nil then attrs
            when false then attrs.merge(column => nil)
            else attrs.merge(column => data)
            end
          end

          define_method("detach_#{name}") do |old_data, new_data|
            uploader.detach(old_data) if old_data && !new_data.nil?
          end
        end
      end

      private

      def flagged(value)
        value == true || value == "1" || value == "true"
      end

      def attachment_errors(error_base, errors, uploader)
        errors.map { i18n.t("#{error_base}.#{it}", **uploader.message_args) }
      end
    end
  end
end

Then finally, here’s a takeout from my app as an example:

# frozen_string_literal: true

module MyApp
  module Operations
    class UpdateAccountDetails < Auth::Operation
      include Deps["i18n", "repos.user_repo"]
      # Include the convenience methods
      include MyApp::Operations::Attachment
 
      # Use the attach macro to declare the attribute and uploader
      attach :avatar, using: Uploaders::AvatarUploader

      def call(account_id:, name:, avatar_upload: nil, cached_avatar: nil, remove_avatar: false)
        user = step find_user(account_id)
        # This is where the actual upload and validation happen
        cached = step attach_avatar(
          upload: avatar_upload,
          cached_json: cached_avatar,
          remove: remove_avatar
        )

        persist(user, name, cached)
        # Deletes the attachment of it's stale or marked for removal
        detach_avatar(user.avatar_data, cached)
      end

      private

      def find_user(account_id)
        user = user_repo.by_account_id(account_id)
        user ? Success(user) : Failure(:not_found)
      end

      def persist(user, name, cached)
        # Merges in the attachment attribute
        user_repo.update_by_account_id(user.account_id, merge_avatar({name:}, cached))
        # Promotes and stores the final attachment data in the store
        store_avatar_data(user, cached)
        # NOTE: We write to the user model twice here because we need the
        # record to be persisted before promoting the attachment, as the
        # database may still raise a constraint error for example.
      end

      def store_avatar_data(user, cached)
        store_data = promote_avatar(cached)
        return if store_data.nil?

        user_repo.update_by_account_id(user.account_id, avatar_data: store_data)
      end
    end
  end
end

In the update action:

# frozen_string_literal: true

module MyApp
  module Actions
    module Account
      module Details
        class Update < MyApp::Action
          include MyApp::AuthenticatedAction
          include Deps[
            "i18n",
            update_details: "operations.update_account_details",
            show_view: "views.account.details.show"
          ]

          before :authenticate!

          params do
            required(:user).hash do
              optional(:name).maybe(:string, max_size?: 120)
              # Register the params attachment here
              optional(:avatar)
              optional(:remove_avatar).maybe(:bool)
            end
          end

          def handle(req, res)
            case outcome(req)
            in Success(_)
              res.redirect_to routes.path(:account), status: 303
            in Failure(:not_found)
              halt :not_found
            in Failure(errors)
              res.status = :unprocessable_entity
              res.render(show_view, **view_data(req, errors))
            end
          end

          private

          def outcome(req)
            return Failure(req.params.errors[:user]) unless req.params.valid?

            user_params = req.params[:user]
            update_details.call(
              account_id: current_user(req).account_id,
              name: user_params[:name],
              # Pass the attachment params to the operation
              avatar_upload: user_params[:avatar],
              remove_avatar: user_params[:remove_avatar]
            )
          end

          def view_data(req, errors)
            {
              rodauth: rodauth(req),
              title: i18n.t("auth.account.details.title"),
              user: current_user(req),
              errors:
            }
          end
        end
      end
    end
  end
end

Does that make sense?

I have to say, this is my first real Hanami app, so I’m still finding my way around the framework after working with Rails for about 20 years. So I’m sure some things can be improved. :slight_smile: But I do have some history with Shrine.

Let me know if anything is unclear.

@wout this was great! it took me a while to read through the code and understand what it was doing, but I was able to simplify it for my own needs and now have images saved as JSON strings in the database :grin:

one more question, if I may be so bold: how do I now use the data after it’s been retrieved from the database? I want it as an attribute in my struct, but is there a better way than just doing a JSON.parse on the data?

Yeah, it was probably a bit overkill to get started, sorry about that :blush:. But great to hear you got uploads working!

As for displaying an uploaded image, the uploader has the uploaded_file class method which takes the stored JSON data. The url_for method in the Shrine plugin I posted uses it:

def url_for(data)
  return unless data

  uploaded_file(data).url
end

In my app I created a part for the User struct to decorate it with an avatar_url helper:

# frozen_string_literal: true

module MyApp
  module Views
    module Parts
      class User < MyApp::Views::Part
        def avatar_url
          # avatar_data here is the JSON from the database
          MyApp::Uploaders::AvatarUploader.url_for(avatar_data)
        end
      end
    end
  end
end

But you could skip the url_for helper and call uploaded_file directly of course:

def avatar_url
  return unless data = avatar_data

  MyApp::Uploaders::AvatarUploader.uploaded_file(data).url
end

Then in your template:

<% if src = user.avatar_url %>
  <%= image_tag(src, alt: user.name) %>
<% end %>

+1 for Shrine! I use it within my application too, storing files locally in dev/test and then on S3 in production.

Then to get the URL helpers, I add them to my structs:

Then I can do image.image_url in my views to get the URL to those images to render them.

thank you both! @wout your solution may have been overkill, but it has helped expose me to more parts of the ecosystem, so I’m thankful for that :blush: