.. _ref_intro_migrations:

==========
Migrations
==========

:index: migrations fill_expr cast_expr

EdgeDB’s baked-in migration system lets you painlessly evolve your schema
throughout the development process. If you want to work along with this guide,
start a new project with :ref:`ref_cli_edgedb_project_init`. This will create a
new instance and some empty schema files to get you started.

.. edb:youtube-embed:: _IUSPBm2xEA


1. Start the ``watch`` command
------------------------------

The easiest way to work with your schema in development is by running ``edgedb
watch``. This long-running task will monitor your schema files and
automatically apply schema changes in your database as you work.

.. code-block:: bash

    $ edgedb watch
    Initialized. Monitoring "/projects/my-edgedb-project".

If you get output similar to the output above, you're ready to get started!


2. Write an initial schema
--------------------------

By convention, your EdgeDB schema is defined inside one or more ``.esdl``
files that live in a directory called ``dbschema`` in the root directory of
your codebase.

.. code-block::

  .
  ├── dbschema
  │   └── default.esdl          # schema file (written by you)
  └── edgedb.toml

The schema itself is written using EdgeDB's schema definition language. Edit
your ``dbschema/default.esdl`` and add the following schema inside your
``module default`` block:

.. code-block:: sdl

  type User {
    required name: str;
  }

  type Post {
    required title: str;
    required author: User;
  }


It's common to keep your entire schema in a single file, and many users use
this ``default.esdl`` that is created when you start a project. However it's
also possible to split their schemas across a number of ``.esdl`` files.

Once you save your initial schema, assuming it is valid, the ``watch`` command
will pick it up and apply it to your database.


3. Edit your schema files
-------------------------

As your application evolves, directly edit your schema files to reflect your
desired data model. Try updating your ``dbschema/default.esdl`` to add a
``Comment`` type:

.. code-block:: sdl-diff

    type User {
      required name: str;
    }

    type Post {
      required title: str;
      required author: User;
    }

  + type Comment {
  +   required content: str;
  + }

When you save your changes, ``watch`` will immediately begin applying your
new schema to the database.

.. note::

    If your schema cannot be applied, the ``watch`` command will generate an
    error. If you're using one of our client bindings as you update your schema
    with ``watch``, you will see the error there the next time you execute a
    query using that client binding.

    If things aren't working the way you expect after making a schema change,
    take a look at the ``watch`` console to find out why.

Once you have the schema the way you want it, and you're ready to lock it in
and commit it to version control, it's time to generate a migration.


4. Generate a migration
-----------------------

To generate a migration that reflects all your changes, run ``edgedb migration
create``.

.. code-block:: bash

  $ edgedb migration create


The CLI reads your schema file and sends it to the active EdgeDB instance. The
instance compares the file's contents to its current schema state and
determines a migration plan.  **The migration plan is generated by the
database itself.**

This plan is then presented to you interactively; each detected schema change
will be individually presented to you for approval. For each prompt, you have
a variety of commands at your disposal. Type ``y`` to approve, ``n`` to
reject, ``q`` to cancel the migration, or ``?`` for a breakdown of some more
advanced options.

.. code-block:: bash

  $ edgedb migration create
  did you create object type 'default::Comment'? [y,n,l,c,b,s,q,?]
  > y
  did you create object type 'default::User'? [y,n,l,c,b,s,q,?]
  > y
  did you create object type 'default::Post'? [y,n,l,c,b,s,q,?]
  > y
  Created dbschema/migrations/00001.edgeql, id: <hash>


.. _ref_intro_migrations_wo_iteration:

Migration without iteration
---------------------------

If you want to change the schema, but you already know exactly what you want to
change and don't need to iterate on your schema — you want to lock in the
migration right away — ``edgedb watch`` might not be the tool you reach for.

Instead, you might use this method:

1. Edit your schema files
2. Create your migration with ``edgedb migration create``
3. Apply your migration with ``edgedb migrate``

Since you're not using ``watch``, the schema changes are not applied when you
save your schema files. As a result, we need to tack an extra step on the end
of the process of applying the migration. That's handled by ``edgedb migrate``.

.. code-block:: bash

  $ edgedb migrate
  Applied m1virjowa... (00002.edgeql)

Once your migration is applied, you'll see the schema changes reflected in your
database.


Data migrations
---------------

Depending on how the schema was changed, data in your database may prevent
EdgeDB from applying your schema changes. Imagine we added a required ``body``
property to our ``Post`` type:

.. code-block:: sdl-diff

    type User {
      required name: str;
    }

    type Post {
      required title: str;
  +   required body: str;
      required author: User;
    }

    type Comment {
      required content: str;
    }

If we hadn't added any ``Post`` objects to our database before this, everything
would have worked fine, but it's likely that, in testing out our schema, we
*did* add a ``Post`` object. It does not have a ``body`` property, but now
we've told the database this property is required on all ``Post`` objects. The
database can't apply this change because existing data would break it.

We have a couple of options here. We could delete all the offending objects.

.. code-block:: edgeql-repl

    db> delete Post;
    {
      default::Post {id: a4a0a40c-d9f5-11ed-8912-1397f7af9fdf},
      default::Post {id: cc051bea-d9f5-11ed-a26d-2b64b6b273a4}
    }

Now, if we save the schema again, ``edgedb watch`` will be able to apply it. If
we have data in here we don't want to lose though, that's not a good option. In
that case, we might drop back to creating and applying the migration outside of
``edgedb watch``.

To start, run ``edgedb migration create``. The interactive plan generator will
ask you for an EdgeQL expression to map the contents of your database to the
new schema.

.. code-block:: bash

  $ edgedb migration create
  did you create property 'body' of object type
  'default::Post'? [y,n,l,c,b,s,q,?]
  > y
  Please specify an expression to populate existing objects in order to make
  property 'body' of object type 'default::Post' required:
  fill_expr>

Because the ``body`` property does not currently exist, the database contains
``Post`` objects without it. The expression you provide will be used to *assign
a body* to any ``Post`` object that doesn't have one. We'll just provide a
simple default: ``'No content'``.

.. code-block::

  fill_expr> 'No content'
  Created dbschema/migrations/00002.edgeql, id:
  m1pjiibv4sa4cao7txpgsbuw2erctmacyrj4qmn45ggapsaztmvxfa

Nice! It accepted our answer and created a new migration file
``00002.edgeql``. Let's see what the newly created ``00002.edgeql`` file
contains.

.. code-block:: edgeql

    CREATE MIGRATION m1pjiibv4sa4cao7txpgsbuw2erctmacyrj4qmn45ggapsaztmvxfa
        ONTO m1nlvzbm7buwktkp4vu4shylq6zp2shruokbbssyeidqmmmfqz77yq
    {
      ALTER TYPE default::Post {
          CREATE REQUIRED PROPERTY body: std::str {
              SET REQUIRED USING ('No content');
          };
      };
    };

We have a ``CREATE MIGRATION`` block containing an ``ALTER TYPE`` statement to
create ``Post.body`` as a ``required`` property. We can see that our fill
expression (``'No content'``) is included directly in the migration file.

Note that we could have provide an *arbitrary EdgeQL expression*! The
following EdgeQL features are often useful:

.. list-table::

  * - ``assert_exists``
    - This is an "escape hatch" function that tells EdgeDB to assume the input
      has *at least* one element.

      .. code-block::

        fill_expr> assert_exists(.body)

      If you provide a ``fill_expr`` like the one above, you must separately
      ensure that all posts have a ``body`` before executing the migration;
      otherwise it will fail.

  * - ``assert_single``
    - This tells EdgeDB to assume the input has *at most* one element. This
      will throw an error if the argument is a set containing more than one
      element. This is useful is you are changing a property from ``multi`` to
      ``single``.

      .. code-block::

        fill_expr> assert_single(.sheep)

  * - type casts
    - Useful when converting a property to a different type.

      .. code-block::

        cast_expr> <bigint>.xp


Further reading
^^^^^^^^^^^^^^^

For guides on advanced migration workflows, refer to the following guides.

- :ref:`Making a property required <ref_migration_names>`
- :ref:`Adding backlinks <ref_migration_backlink>`
- :ref:`Changing the type of a property <ref_migration_proptype>`
- :ref:`Changing a property to a link <ref_migration_proptolink>`
- :ref:`Adding a required link <ref_migration_reqlink>`

For more information on how migrations work in EdgeDB, check out the :ref:`CLI
reference <ref_cli_edgedb_migration>` or the `Beta 1 blog post
</blog/edgedb-1-0-beta-1-sirius#built-in-database-migrations-in-use>`_, which
describes the design of the migration system.
