Check your version

This video covers a topic in Drupal 7 which may or may not be the version you're using. We're keeping this tutorial online as a courtesy to users of Drupal 7, but we consider it archived.

Introduction to the Migrate Module Series

Video loading...

  • 0:04
    Migrate Module Basics for Drupal 7 Series Introduction to the Migrate Module
  • 0:06
    with Joe Shindelar
  • 0:18
    In this series, we're going
  • 0:20
    to take a look at the Migrate module for Drupal 7.
  • 0:24
    The Migrate module is a tool which allows
  • 0:26
    us to import data into Drupal.
  • 0:28
    It's actually a really robust framework
  • 0:30
    for writing our own custom migrations,
  • 0:32
    extracting some source data, importing it into Drupal as nodes,
  • 0:37
    or users, or whatever it is that we need to import,
  • 0:39
    and then transforming it along the way, if necessary.
  • 0:43
    Throughout the course of the series, we're
  • 0:45
    going to be taking a look at all of the different functionalities
  • 0:47
    that the migrate module provides.
  • 0:49
    We'll start by doing a tour of the user interface looking at how we
  • 0:53
    can import things into Drupal by looking at the examples that
  • 0:56
    are provided with the migrate module.
  • 0:59
    The examples are really great, and have some great comment
  • 1:01
    and great documentation, and are a great place
  • 1:04
    to find out more about how migrations work.
  • 1:07
    After we've had a chance to look at the migrations and how those work,
  • 1:11
    we're going to take a look at writing our own migrations.
  • 1:14
    We're going to need some source data to import into Drupal,
  • 1:17
    so we found a great database that contains information about baseball
  • 1:21
    players and teams and the ability to link a player to a team
  • 1:24
    so we know which players played for which teams.
  • 1:27
    Now, there's a bunch of additional information in there, as well.
  • 1:30
    We'll talk about that information, how it translates to the content
  • 1:34
    types that we're going to build in Drupal.
  • 1:36
    And in some cases, how it doesn't quite translate.
  • 1:38
    And we need to use the Migrate module to transform the data
  • 1:41
    during the migration so that our source that ends up matching what
  • 1:45
    we need it to in the resulting destination.
  • 1:49
    Of course, we'll talk about what a source is
  • 1:51
    and what a destination is, and a little bit about the extract,
  • 1:55
    transform, and load process that the migrate module uses in order
  • 1:59
    to pull records of data out of whatever source you provide for it,
  • 2:03
    whether that's a MySQL database, or a CSV file, or maybe a JSON feed,
  • 2:09
    transform it doing things like converting inches to feet
  • 2:13
    or concatenating multiple strings together.
  • 2:16
    And then saving it into a destination.
  • 2:19
    And that destination might be a node or a taxonomy term.
  • 2:23
    We'll talk about how the migrate module is
  • 2:25
    aware of the different data types within Drupal,
  • 2:28
    and as a result, makes it really easy for us to do things
  • 2:31
    like configure fields via the field API,
  • 2:34
    have the Migrate module know about those fields,
  • 2:37
    and then allow us to map source data to a specific field
  • 2:42
    using the migrate module.
  • 2:44
    We're also going to take a look at populating entity reference fields
  • 2:47
    using the migrate module and the things
  • 2:49
    that we need to do in order to link up a record of source data
  • 2:53
    with the node that got created in Drupal
  • 2:56
    as a result of the migration.
  • 2:58
    This is another one of the really handy features that the Migrate
  • 3:01
    module provides us, the ability to translate those unique IDs so
  • 3:05
    that we can look up based on a mapping, which source
  • 3:08
    record maps to which destination record.
  • 3:11
    We're also going to take an in depth look at field mappings.
  • 3:15
    We'll actually dedicate a couple of lessons
  • 3:17
    to this, since it's one of the most important concepts
  • 3:19
    in the migrate module, the ability to take a column from your database
  • 3:23
    that says, this is the player name, and then map it to a destination
  • 3:28
    in Drupal, something like the title of a node,
  • 3:30
    creating this link between a piece of source data
  • 3:34
    and what actually gets saved on the destination side.
  • 3:37
    Field mappings also allow us to do some transformation
  • 3:40
    in between if we need to.
  • 3:42
    We're going to have an example where we take the first name
  • 3:45
    and last name field from our source data
  • 3:47
    and concatenate those things together so that we have
  • 3:50
    a single string of data which is saved
  • 3:52
    as the title of a node on the Drupal side.
  • 3:55
    In addition to allowing us to write the code that creates a migration,
  • 3:59
    the Migrate module also provides a lot of code
  • 4:01
    that supports running those migrations.
  • 4:03
    So we'll take a look at doing that.
  • 4:04
    How can we run migrations via the user interface, and also,
  • 4:08
    how can we run migrations with Drush?
  • 4:10
    Then we'll use Drush to run a bunch of migrations, import records
  • 4:14
    into Drupal.
  • 4:15
    We'll also roll those migrations back,
  • 4:17
    allowing us to say, do something like import 10 players,
  • 4:21
    see that one of them failed, decide that we need to roll it back,
  • 4:24
    and delete the players that we just imported.
  • 4:26
    Make a couple of changes to our migration code,
  • 4:29
    and then run it again.
  • 4:30
    It's a pretty common practice, the need
  • 4:32
    to try to run these migrations more than once,
  • 4:34
    and debug the problems in either your code, or your source data,
  • 4:38
    or maybe your destination.
  • 4:40
    Luckily for us, the Migrate module makes this pretty easy to do.
  • 4:44
    We'll spend a little bit of time talking about how the Migrate
  • 4:47
    module allows us to collaborate with our other team members
  • 4:50
    when discussing field mappings, linking a field mapping to an issue
  • 4:54
    in our ticket tracking system so that we can discuss what the end
  • 4:58
    value should really be and how this mapping should work.
  • 5:01
    Sometimes with migrations, it's important for people that aren't
  • 5:04
    necessarily PHP developers to be able to understand what's going on.
  • 5:09
    They ultimately know the data from the source
  • 5:11
    and what it's supposed to be in the destination.
  • 5:13
    And a lot of times, we just need to help them figure out once they've
  • 5:17
    drawn the links between the two fields,
  • 5:19
    how to write the code to make that happen.
  • 5:22
    Finally, we'll end the series with a lesson that discusses some
  • 5:26
    of the potential problems that you'll run into when trying to run
  • 5:29
    migrations in ways that we can try to solve those.
  • 5:32
    The migrate module does provide a lot of features
  • 5:35
    for doing this thing of running an import,
  • 5:37
    testing things out, rolling it back, making some changes,
  • 5:40
    and repeating that process over and over.
  • 5:43
    It also provides some functions for logging messages into the database
  • 5:47
    so that if a record fails during the migration,
  • 5:50
    even if the migration took 20 minutes
  • 5:52
    and we left while it was running, we can see why it failed when we come
  • 5:56
    back and work on trying to fix that.
  • 5:59
    So let's go ahead and get started learning about using
  • 6:01
    the migrate module and importing data from various different sources
  • 6:05
    into Drupal.

Introduction to the Migrate Module Series


This series is focused on using the Migrate module to import data that exists in various different sources into a Drupal 7 website. The Migrate module provides an extremely flexible and robust framework for accessing data from various sources and importing or migrating that data to Drupal. With built in support for creating core Drupal data types likes nodes, users, and taxonomy terms, the Migrate module is one of the best solutions available for importing content into Drupal.

This series kicks off with Joe Shindelar explaining the basic components that make up a data migration, and the terminology and code that is specific to the Migrate module. Then continues with a series of lessons that take you from installing the Migrate module to writing and running your own custom data migration.

Throughout the series Joe teaches us how to run a data migration using both the Migrate module's UI and drush, and some of the plusses and minuses of both methods. Joe also talks about the various different sources, or types of data, from which the Migrate module can read data and how to map the unique fields in a row of source data to their corresponding Drupal content types and fields.

By migrating from a single source into two different Drupal content types we'll also have the opportunity to learn about creating relationships during a migration and mapping the resulting information to an entity reference field. During the course of writing a custom migration Joe will show us how and where we can add code to perform additional runtime data munging during our import process. We'll learn about importing data into multi-value fields, and even providing defaults for fields that don't have information. Then we'll look at some of the tools the migrate module provides for collaborating with team members in order to create a successful migration path.

Finally we'll wrap up the series by looking at a couple of different techniques for debugging our migrations and dealing with pesky source material that just doesn't want to be imported.

Because this series is focused primarily on writing custom data migrations, and since the Migrate module itself requires at least some amount of code to be written to perform a migration, it is suggested that students be familiar with PHP and basic Object Oriented Programming techniques. Although not required to run a migration, Joe uses the drush command line tool extensively in this series. If you need a refresher on using drush take a moment to watch our drush series.

Additional resources: