Introduction to Ecto.Multi

Ecto.Multi docs
Helpful guide from Elixir School!

We recently had some internal discussions about Ecto.Multis. One co-worker claimed to have never really understood them, another wrote an advanced blog post on them! I felt somewhere in between. So I thought I’d put together a primer outlining my understanding and exploring other aspects I was less familiar with from a high level.

What is an Ecto.Multi?

A built-in tool that strings together database changes.

A Multi is a data structure provided by Ecto that acts as a single unit, a compound function, putting several sub-functions together as one. These are often used for database transactions (changes) but can include anything you might want.

The uniqueness of a Multi is that it is a single unit, all of the parts included therein fail or succeed as one. All or nothing.

Why they are hard

It’s a lot to look at. They can be confusing and long. Like this one.

Knowing the structure of a Multi helps, and extracting the inner workings in each part allows zooming out to understand the whole to be much easier.

When to use them

A Multi is usually set up around a process.

Creating a new user?

• need to create them in a users table?
• create their role in a roles table?
• add an entry into a log?
• alert an admin?
• and send them an email?

A Multi is helpful in a process like this because if something goes wrong along the way, the entire action is abandoned. That means stopped and any database changes rolled back.

If the role is unable to be created, the log entry or anything that follows will not be entered and the user created will be undone.

It’s also a key tool because a database record can be created, then it can be used for other things.

Create a user, then grab the user id to create a role record, log entry etc.

When not to use it

If you are not making changes to the DB it’s not necessary. Multis are a part of Ecto, meaning that they are inherently meant to be used for database transactions. And the innate rollback feature is a key part of it.

If you are not making any database changes, use a with statement. See here for more info on with statements.

What would it look like if we didn’t have Multis?

case statements
with statements
Repo.transactions with rollbacks

Most likely, you’d have a bunch of case or with statements wrapped in a transaction with rollback at the end or on each portion of the transaction in case it goes wrong

simple example

Repo.transaction(fn ->
  with {:ok, _first} <- create_record(params),
       {:ok, _second} <- create_record(params) do
    :ok
  else
    {:error, changeset} ->
      changeset   
      |> Repo.rollback()
  end
end)

as a multi

Ecto.Multi.new()
|> Ecto.Multi.insert(:record_1, User.create_changeset(%User{}, params))
|> Ecto.Multi.insert(:record_2, Role.create_changeset(%Role{}, params))
|> Repo.transaction()

Types

I think of Ecto.Multis as having two categories

Database Functions:

delete/4
delete_all/4
insert/4
insert_all/5
insert_or_update/4
update/4
update_all/5

Non-database functions:

new/0
append/2
error/3
merge/2
merge/4
prepend/2
run/3
run/5
to_list/1

Basic Structure

Ecto.Multi.new()
|> ...your multis here...
|> MyApp.Repo.transaction()

always start with

Ecto.Multi.new()

ends with a repo transaction

MyApp.Repo.transaction()

Database Functions

Most of these are straight forward as their names indicate. There are single and multiple versions for for one or many changes at once. These differences also cause the arguments to change.

Let’s take a look at arguments for each type

Single Transactions

insert - (multi, name, changeset_or_struct_or_fun, opts \\ [])
delete - (multi, name, changeset_or_struct_fun, opts \\ [])
update - (multi, name, changeset_or_fun, opts \\ [])
insert_or_update - (multi, name, changeset_or_fun, opts \\ [])

Argument Breakdown

Note that update functions do not take structs. That’s because they start with an existing DB record rather than a blank slate.

multi
the multi itself or previous operations in the sequence, usually used as the first argument and piped in, so you’ll only see it at the top and it won’t appear as an argument in any of the functions in the pipe chain

Ecto.Multi.new()
|> Ecto.Multi.insert(:name, changeset_or_struct_or_fun, opts \\ [])

name
custom name for the current operation. Each name must be unique from other in the Multi.

Ecto.Multi.new()
|> Ecto.Multi.insert(:user, changeset_or_struct_or_fun, opts \\ [])

changeset_or_struct_or_fun
body; the action taking place in that step

changeset

• eg User.create_changeset(%User{}, params)

Ecto.Multi.new()
|> Ecto.Multi.insert(:user, User.create_changeset(%User{}, params), opts \\ [])

struct

• eg %Post{title: "first"}

Ecto.Multi.new()
|> Ecto.Multi.insert(:post, %Post{title: "first"}, opts \\ [])

function

• eg:

 fn %{op_one: op_one} ->
  Comment.create_changeset(%Comment{},
  %{title: op_one.title})
 end),

Ecto.Multi.new()
|> Ecto.Multi.insert(:post,  fn %{op_one: op_one} ->
  Comment.create_changeset(%Comment{},
  %{title: op_one.title})
 end), opts \\ [])

changeset_or_fun
same as the previous changeset_or_struct_or_fun, but update functions cannot take a struct

Multiple transactions

insert_all - (multi, name, schema_or_source, entries_or_fun, opts \\ [])
delete_all - (multi, name, queryable_or_fun, opts \\ [])
update_all - (multi, name, queryable_or_fun, updates, opts \\ [])

Argument Breakdown

schema_or_source
What schema is to be referenced for these changes

• eg User or Post

Ecto.Multi.new()
|> Ecto.Multi.insert_all(:user, User, entries_or_fun, opts \\ [])

entries_or_fun
What are the changes? A function that builds changes to go in or a list if changes

• eg  [%{name: "Sarita"}, %{name: "Wendy"}]

Ecto.Multi.new()
|> Ecto.Multi.insert_all(:user, User, [%{name: "Sarita"}, %{name: "Wendy"}], opts \\ [])

queryable_or_fun
Grab existing records from the database to manipulate

• eg queryable = from(p in Post, where: p.id < 5)

queryable = from(p in Post, where: p.id < 5)
Ecto.Multi.new()
|> Ecto.Multi.delete_all(:delete_all, queryable)
|> MyApp.Repo.transaction()

A queryable would be a raw sql statement that specifies the records to perform the action on.
eg queryable = from(Movie, where: [title: "Ready Player One"])

If using a function, it must return a queryable.

Non-database functions

Here’s the list again and what they do.

new/0 - start a new Multi
append/2 - string together two Multis (adding at the end)
error/3 - cause failure for a given value
merge/2 - add in another multi via another function (usually on a condition)
merge/4 - add in another action using a module directly
prepend/2 - string together two multis (adding before)
run/3 - add in a function to the sequence (not a DB transaction)
run/5 - add in a function to the sequence using a module function directly
to_list/1 - returns a list of the operations in a multi (given names)

These methods allow for flexibility when it comes to use cases. Add in other database transactions or couple multis together based on a condition. Merge in a reduce function that uses a new multi for each element in a list allowing them to fail apart from the greater scheme if desired.

Arguments in these methods are often structured a little differently.

1  :name, fn, repo, changes -
2   ..whatever function…
3   {:return, tuple}
4  end)

line 1:

• :name - name
• fn - embedded function
• repo - Ecto.Repo (access to performing database changes)
• changes - changes made in the multi sequence up to that point

line 2: whatever functionality needed to be done

line 3: return should be a tuple

• eg {:ok, result}

a multi is automatically returned

Ecto.Multi.run()

Add in an action that makes use of changes made before it in the multi. Returns a multi automatically.

Pattern match on changes to create a variable or pull out a particular item.

example using changes

|> Ecto.Multi.run(
  :image_processing,
  fn _repo, %{image: image} ->
    result = do_something(image)
    {:ok, result}
  end)
)

example using repo

|> Ecto.Multi.run(
  :name,
  fn repo, _changes ->
    {:ok, image} = Images.get(image_id)
    repo.update(
      Image.processed_image_changeset(image)
    )
  end)
)

Ecto.Multi.merge()

Adds another multi into the fold.

Like .run() it adds in more actions, but in this case it might be a stand-alone Multi used in other places as well or simply have multiple actions included within.

arguments are just a function (fn) and changes (%{image: image})

|> Ecto.Multi.merge(
  fn %{new_user: new_user, current_user: current_user} ->
    audit_log(new_user, current_user)
  end)
)

def audit_log(new_user, current_user) do
  Ecto.Multi.new()
  |> Ecto.Multi.insert(
      :create_user, 
      Audit.changeset(current_user, %{action: "created new user", target: new_user})
  )
  |> Ecto.Multi.insert(:new_user, Audit.changeset(new_user, %{action: "user created}))
end

Cool Features

Passing along all the newly made changes
With each step in the sequence, a variable is created (based on :name) that represents anything returned in that step. This variable can be used for actions down the line in the sequence.

For example, use the id from your newly created user to create a role with that id.

You can also pattern match on all the changes that have come up to that point, making it easy to grab only the things you need.

1    Ecto.Multi.new()
2    |> Ecto.Multi.insert(:user, User.create_changeset(%User{}, admin_params))
3    |> Ecto.Multi.insert(:role, fn %{user: user} ->
4      Role.create_changeset(%Role{}, %{
5        type: "admin",
6       user_id: user.id,
7        registration_site_id: registration_site.id
8      })
9   end)
10   |> Ecto.Multi.run(:welcome, fn _repo, %{user: user} ->
11     Users.send_welcome_email(user)
12     {:ok, user}
13   end)

In the above, on line 3, user is the only thing that’s been created so far, but I can still pattern match it and give it a variable name since I need it.

On line 10, I only need the user.

But I could have just as easily grabbed both the user and role and made variables for them to use: %{user: user, role: role}

Add in any function
You can also throw in a regular ol’ function if you want to. If so, the function must take the multi as an argument and returns the multi as well to pass it down the line. Usually this is done by having another multi inside that regular function which would return the multi automatically.

Primary use case would be to present cleaner code by abstracting certain parts.
Throw in a function that holds different sections of the multi and give it a descriptive name.

Ecto.Multi.new()
|> logger()
|> get_useful_information()
|> Ecto.Multi.update(:new_info, fn %{info: info} ->
  User.update_changeset(%User{},
  %{preference: info.preference}
end) 
|> Repo.transaction()

def logger(multi) do
  # These will run out of order from the chain
  # They will *not* run inside of the database transaction
  # You can consider this at "compile time" for the multi
  Logger.info("Logging this thing not part of the transaction")

  multi                                                                                                                                                                   
end

def get_useful_information(multi) do
  Ecto.Multi.run(multi, :get_useful_information, fn _ ->
    Something.get(id)
  end)
  multi 
end

What they look like all put together

Use cases

Delete multiple things at once

Ecto.Multi.new()
|> Ecto.Multi.delete_all(:staged_changes, Ecto.assoc(struct, :staged_changes))
|> Repo.transaction()

example in context: here

Create an avatar when a user is created, if they have selected one

Ecto.Multi.new()
|> Ecto.Multi.insert(:user, changeset)
|> Ecto.Multi.run(:avatar, fn _repo, %{user: user} ->
  Avatar.maybe_upload_avatar(user, params)
end)
|> Repo.transaction()

Update a program

• line 2 - with any changes in to items in the schema
• line 3 - add partners
• line 4 - add owners
• lines 5-7 - create events?
• lines 8-10 - add a logo?
• lines 11-13 - add a banner image?
• line 14 - log who is making changes in a security log, what kind of changes and their ip

1    Ecto.Multi.new()
2    |> Ecto.Multi.update(:program, changeset)
3    |> attach_partners(params)
4    |> attach_program_owners(params)
5    |> Ecto.Multi.run(:event, fn _repo, %{program: program} ->
6      	maybe_create_event(program, changeset)
7    end)
8    |> Ecto.Multi.run(:logo, fn _repo, %{program: program} ->
9      Logo.maybe_upload_logo(program, params)
10   end)
11   |> Ecto.Multi.run(:banner_image, fn _repo, %{program: program} ->
12     BannerImage.maybe_upload_banner_image(program, params)
13   end)
14   |> add_to_security_log_multi(current_user, "update", remote_ip)
15   |> Repo.transaction()

In a pipeline - query then update

def set_statuses() do
  Program
  |> where([p], p.status == "ready")
  |> Repo.all()
  |> Enum.reduce(Ecto.Multi.new(), fn program, multi ->
    Ecto.Multi.update(multi, {:program, program.id}, set_status(program))
  end)
  |> Repo.transaction()
end

Conclusion

And there you have it! This post is meant to cover a wide range of information on Multis, answering the whys, whens, whats and hows. It does not have examples for every Multi method but I’m looking forward to finding more reasons to use the lesser seen ones in the future. This is just the beginning! There are certainly patterns out there that make these even more tailored to specific use cases, check out our advanced post to see more.

Header photo by Pierre Châtel-Innocenti on Unsplash