Whether you're updating from Drupal 6 or Drupal 7, or importing data from some other source, you need to know about the migrate system in Drupal core. This guide provides a comprehensive look at the features of the Drupal migrate API for both Drupal-to-Drupal updates, and migrating from any external datastore into Drupal.
This guide is for anyone who wants to:
- Update to the latest version Drupal from Drupal 7 or earlier
- Import data from external datasources including MySQL, CSV, JSON, or XML into Drupal 9 or 10
- Provide an update path from Drupal 6 or 7 for a contributed module
Where should I start?
If you're new to the Migrate API, start from the beginning and get an overview of how the system works and how all the pieces fit together. This is essential background information for anyone wanting to migrate data into Drupal. Additionally, part of performing any migration is planning and preparation. Migrations involve preparing and analyzing your source data, building a new website that data can be migrated into, and lots of testing, rolling back, and testing again, in order to get everything right.
Update or migrate?
The Drupal core migration system is used for both importing data from custom sources, and for migrating content and configuration from previous versions of Drupal into Drupal 9 or 10 -- effectively updating your site from one major version to another. Depending on which of these two tasks you're performing you'll use the tools in different ways.
Updating from Drupal 7 or earlier
If you're migrating from old versions of Drupal, start with our Preparing for a Drupal-to-Drupal Migration tutorial. From there you'll learn how to perform a basic update using either the built in UI or the contributed Drush commands, as well as some common best practices to help things go as smoothly as possible. If you're looking to customize your Drupal-to-Drupal migration a bit, continue through the tutorials on writing custom migrations, and then we'll go in-depth on how to use those skills to tailor the Drupal-to-Drupal update process to your specific use-case.
Importing data from external sources
If you're importing data from a non-Drupal datastore, start with Set up Migrate Demo Site and Source Data . We'll walk through the process of connecting the migrate system to an external data source, writing custom migration paths, using custom process plugins to transform data during import, and best practices for executing your custom migrations. We'll primarily look at using an external MySQL database as our data source, but the techniques learned will apply to any data source. We'll also discuss how to extract data from CSV, JSON, and XML sources.
The Drupal migration system is currently a work-in-progress; there are still some kinks, and best practices continue to evolve. As with all of our Drupal material, we're committed to keeping these tutorials up-to-date as work progresses. We'll keep an eye on the changes that are happening both in core and with the main contributed modules and make adjustments as needed.
Whether you're updating from Drupal 6 or Drupal 7, or importing data from some other source, you need to know about the migrate system in the latest version of Drupal. This tutorial provides an overview and links to additional tutorials where you can learn more about how all the individual parts work.
By the end of this tutorial you should have a better understanding of what the migration system is capable of and know where to find more information about how to use it.
Planning for a migration is essential. In our collective experience we've never once seen someone sit down and execute a migration flawlessly on their first attempt. Migrations involve preparing and analyzing your source data, building a new website that data can be migrated into, and lots of testing, rolling back, and testing again, in order to get everything right. By the end of this tutorial you should be ready to start planning for your own Drupal migration project, and have a better understanding of the challenges of migrating from one system to another.
To follow along with the rest of the migration tutorials you'll want to make sure you understand the following concepts and terms as they relate to Drupal migrations.
In this tutorial, we'll take look at the basic components of a migration and familiarize ourselves with some of the concepts and terminology needed to understand how the system works. We'll cover:
- What is a migration?
- Migration templates
- The extract, transform, load process
- Destinations and sources
- Additional Drupalisms
By the end of this tutorial you should be able to identify the various components that a migration is composed of, and explain at a basic level what each is responsible for.
Every migration is unique, which means there are loads of great examples you can review and learn from. We'll keep this tutorial up-to-date with our favorites and let you know how we think these examples might help.
In Drupal, there are 3 modules in core related to migration that you'll want to know about. These modules can help you import data into Drupal from disparate sources, or upgrade from a previous version of Drupal.
In this tutorial we'll look at what each of these core migration modules do, and talk about when you'll need to use them.
There is a whole ecosystem of contributed modules that build on the Migrate API in Drupal core. They do things like provide Drush commands for working with migrations, add new sources (CSV, JSON, etc.), add new destinations, provide code examples, and fill in other gaps.
In this tutorial we'll look at some of the most commonly used contributed modules and what they do.
Before you begin a Drupal 6 or 7 (source) to Drupal 8 or 9 (destination) migration there are a number of things you should consider. Taking the time to plan your migration will help to ensure that you're successful. In this tutorial we'll take a high-level look at:
- Evaluating your existing Drupal 6/7 site for migration feasibility
- Preparing your source Drupal 6/7 site for a migration
- Preparing the destination Drupal site you're migrating to
By the end of this tutorial you should be ready to start assessing the feasibility of performing a successful migration, and begin making a migration plan.
When preparing to migrate from an older version of Drupal (Drupal 7 or previous) to the latest version of Drupal (Drupal 8 or later) you'll need to determine if the contributed modules you use are ready to go. The Upgrade Status module can give you a list of all the modules installed on your site, and information about the availability of a latest Drupal version. It's not perfect, and will still require some manual research for some modules, but it is a great start towards helping you plan for your migration.
In this tutorial we'll:
- Install the Upgrade Status module and use it to generate a status report
- Review the generated report
- Talk about using the Contrib Tracker project, and other methods for getting more details about the status of upgrades for any module
By the end of this tutorial you should be able to evaluate all the installed modules on your existing Drupal site and determine the status of a module release that is compatible with the latest version of Drupal.
It's not uncommon to patch, or customize, contributed modules during the lifetime of a site. It's important to know if you've done so when planning for a migration since you'll want to be sure that you don't lose any customizations when you start using an updated version of a module.
In this tutorial we will walk through using the Hacked! module to check for any alterations to your Drupal code base.
The Drupal-to-Drupal migration system is still a work in progress. As such, there are a few things that simply don't work, and a few others that still have kinks to be ironed out. In this tutorial, we'll look at some of the common hang-ups that we've encountered, the status of resolving those issues, and what your options are in the meantime.
The Migrate Drupal UI module allows you to execute a migration from older versions of Drupal to the latest version of Drupal. In this tutorial we'll:
- Run a full Drupal 6 to latest-Drupal (Drupal 8 or 9) Migration from the UI
- Explore the user interface as it exists
- Understand how we can deal with the output from our migration
In this tutorial we will run a site migration using Drush, and understand how to deal with any failures that occur.
Before we can learn to write a custom migration, we need some sample data and a destination site for that data.
In this tutorial we'll obtain some source data to work with and configure our Drupal destination site by creating the necessary content types and fields to accommodate the source data. Then we'll look at the data that we'll be importing and start to formulate a migration plan.
By the end of this tutorial you'll have some source data and an empty but configured destination Drupal site ready for data import.
Source plugins extract data from a source and return it as a set of rows representing an individual item to import and some additional information about the properties that make up that row.
Anyone writing a custom migration, or module developers who want to provide a migration path from their Drupal 6 or 7 module, will need to work with source plugins.
In this tutorial we'll talk about the role that source plugins fulfill and how they work. By the end of this tutorial you should be able to determine whether or not you need to write a source plugin for your migration.
This tutorial covers writing a custom source plugin that imports data from a MySQL database into Drupal nodes. After completing this tutorial you should understand how to write your own custom source plugin that can:
- Extract data from an SQL source
- Describe the various fields in the source data to the Migrate API for mapping
- Provide unique IDs for each row of imported data
By the end of this tutorial you should be able write a custom source plugin that uses an SQL data store as well as have a foundation for writing source plugins that extract data from any source that PHP can read.
Process plugins manipulate data during the transform phase of the ETL process while the data is being moved from the source to the destination. Drupal core provides a handful of common process plugins that can be used to perform the majority of data transformation tasks. If you need some functionality beyond what is already provided you can write your own custom process plugins.
In this tutorial we'll:
- Examine the role that process plugins fullfil
- Understand the processing pipeline
- List the existing process plugins in Drupal core and what each one does
- Better understand when you might need to write your own process plugin
By the end of this tutorial you should be able to explain what process plugins do, and understand how you'll make use of them in your own migration.
This tutorial covers writing a custom process plugin that you can use to manipulate the value of any field during the process (or transform) phase of a migration. Process plugins take an individual field value provided by a source plugin, and perform transformations on that data before passing it along to the load phase.
In this tutorial we'll write a process plugin that can either uppercase an entire string or the first letter of each word in the string depending on configuration.
By the end of this tutorial you should know how to start writing your own process plugins.
Destination plugins handle the load phase of the ETL (Extract, Transform, Load) process and are responsible for saving new data as Drupal content or configuration.
In this tutorial, we'll:
- Examine the role that destination plugins fulfill
- Learn about existing destination plugins
- Better understand when you might need to write your own destination plugin
By the end of this tutorial, you should be able to explain what destination plugins does and understand how you'll make use of them in your own migration.
Migration plugins are the glue that binds a source, a destination, and multiple process plugins together to move data from one place to another. Often referred to as migration templates, or simply migrations, migration plugins are YAML files that explain to Drupal where to get the data, how to process it, and where to save the resulting object.
Source, process, and destination plugins do the heavy lifting in each phase of the ETL process in a custom migration. We need to choose which plugins we want to use for each phase, as well as map fields from our source data to fields at our destination. A migration YAML file glues it all together and gives it a unique name that we can use to run it.
In this tutorial we'll:
- Determine what information we're going to move, and where we're going to move it to
- Install Migrate Plus and Migrate Tools which we'll use to run our custom migration
- Write a custom migration plugin (configuration) YAML file that will work with Migrate Tools
By the end of this tutorial you should be able to write a custom migration YAML file and understand how to choose the source, destination, and process plugins that will do the work.
When running migrations you can use the
highwater_mark source plugin configuration option to influence which rows are considered for import on subsequent migration runs. This allows you to do things like only look at new rows added to a large dataset. Or to reimport records that have changed since the last time the migration was run. The term, highwater mark, comes from water line marks found on structures in areas where water level changes are common. In running migrations, you can think of a highwater mark as a line that denotes how far the migration has progressed, and saying, "from now one, we only care about data created after this line".
Another common use case for highwater marks is when you're importing a large dataset and the system runs out of resources. Usually this will look like a migration failing because it timed out, or the process ran out of memory. A highwater mark should allow you to pickup from where you left off.
In this tutorial we'll:
- Define what a highwater mark is, and how you can use them to limit the rows considered for importing each time a migration is executed.
- Demonstrate how highwater marks can be used to reimport source records that have been modified since the previous time the migration was executed.
- Introduce the
By the end of this tutorial you should be able to define what a
highwater_mark is and how to use them to speed up the import of large datasets or force the migration to reimport records when the source data is changed.
You should also be aware of the
track_changes feature which is a slower, but more dynamic, method of checking for changes in source data and reimporting records when a change is found.
If you need to write a migration that is capable of being executed multiple times and picking up changes to previously imported data, you can use the
track_changes configuration option of most source plugins. This will tell the migration to keep a hash of each source row, and on subsequent runs of the same migration it will compare the previous hash to the current one. If they don't match this means the source data has changed, and Drupal will reimport that record and update the existing destination record with new data.
track_changes differs from calling
drush migrate:import --update in that using
--update will force every record to be re-imported regardless of whether the source data has changed or not.
In this tutorial we'll:
- Learn how change tracking works to detect changes in your source data
- Use the
track_changesoption in a migration
Note that progress of a migration can also be tracked using highwater marks if the source data has something like a
last_updated timestamp column. Using highwater marks in this case is likely more efficient than using
track_changes. It's a good idea to understand how both features work, and then choose the appropriate one for each migration.
As of right now, the most reliable way to run custom migrations is using Drush. Depending on the version of Drush you're using you may also need the Migrate Tools module. In this tutorial we'll walk through using Drush to run a custom migration, as well as the other commands that can be used to manage the execution of migrations.
By the end of this tutorial you should know how to run your custom migrations.
Drupal core provides support for migrating data from Drupal 6 or 7. The templates provided in core migrate your data in a very specific way. They attempt to copy things verbatim whenever possible. However, you may wish to alter this for your migrations. In this tutorial we are going to explore the various ways that you can alter the existing Drupal-to-Drupal migrations to meet your needs.
Why use Migrate Upgrade and Migrate Plus? The short answer is: because it'll save you from having to type out a bunch of YAML by scaffolding a migration for you.
The contributed Migrate Plus and Migrate Upgrade modules are commonly used to aid in the process of performing a Drupal-to-Drupal migration. The combination of the two makes it easier to customize the configuration of individual migrations on a per-project basis by allowing you to edit each individual migrations configuration using the same mechanics that you would to write a custom migration.
It's not the only approach to making these changes, but for many people (including us), it's the most efficient way.
In this tutorial we'll:
- Get a more in-depth look at the role of the Migrate Plus and Migrate Upgrade modules
- Discuss the use cases for using them as part of a Drupal-to-Drupal migration
By the end of this tutorial you should be able to explain what both the Migrate Plus and Migrate Upgrade modules are used for.
One method of creating a custom Drupal-to-Drupal migration involves using the Migrate Upgrade module to generate a set of migration configuration entities that you can use as a starting point. Even if you're not going to use the generated output in the end, this is still an informative exercise as it allows you to see examples of various migration paths.
In this tutorial we'll:
- Use the Migrate Upgrade module's Drush commands to import Drupal core's migration templates
- Examine the generated configuration entities
- Use the Migrate Tools module's Drush commands to view a list of the individual migrations that make up our Drupal-to-Drupal migration
By the end of this tutorial, you should be able to use the Migrate Upgrade module to generate the migrations that Drupal core would use to migrate content, list those migrations, and inspect them individually.
If you want to modify the Drupal-to-Drupal migrations created by Migrate Upgrade you'll need to export the Migrate Plus configuration entities, convert the ones you want to customize to standard migration .yml files, and put them into a custom module. Then, you can make edits the YAML definition of the migration, and keep your customizations in Git.
We recommend creating a new module to house the code that makes up your custom Drupal-to-Drupal migration.
In this tutorial we'll:
- Create a new module
- Export the migration configuration entities generated by
drush migrate-upgradeas YAML files
- Copy the files that represent the migrations we're interested in into our new module
- Customize the copied files
By the end of this tutorial you should have a new module that contains the starting migration YAML files for your custom migration.
When working on configuration in a module, whether as part of a migration that uses Migrate Plus configuration entities, or while developing custom configuration entities, you'll often need to re-import the configuration stored in the .yml files of the modules config/install/ or config/optional/ directories. This is tricky though, because Drupal only reads in those default configuration settings when the module is first enabled. So any changes you make to those files after the module has been installed will not be reflected without these workarounds.
Knowing how to do this can improve the developer experience of adding (or debugging) the default configuration that's provided with a module. Or for anyone using Migrate Plus configuration entities as part of a migration.
In this tutorial we'll:
- Learn about the Configuration Development module
- Look at how you can use Drush to perform a partial configuration import
- Write an implementation of
hook_uninstall()to remove a module's configuration when it's uninstalled
By the end of this tutorial you should be able to re-import the configuration provided by a module without having to uninstall and then reinstall the module.
As part of creating a custom Drupal-to-Drupal migration we want to limit the set of users that are migrated from our source site into our destination Drupal site. In this tutorial we'll:
- Extend the existing source plugin
- Alter the query that's used to select users from our source site
- Update our user migration to use the new source plugin
By the end of this tutorial you should be able to override the core source plugins used when migrating from prior versions of Drupal in order to gain more control over exactly what is migrated.
Every row returned, from every source, during the execution of a migration is passed through
hook_migrate_MIGRATION_ID_prepare_row(). Using these two hooks we can augment our migration in a variety of different ways.
In this tutorial we'll:
- Discuss the use case for
hook_migrate_prepare_row()and use it to skip all but a select list of fields during the field migration
hook_migrate_MIGRATION_ID_prepare_row()and use it to skip all but a select list of node types
By the end of this tutorial you should have a better understanding of when
hook_migrate_prepare_row() might be useful when writing your own migration, as well as how to skip rows in a migration based on conditional logic.