Custom Drupal-to-Drupal Migrations with Migrate Tools

Image
On the Drupal Migration Trail

Editor's note: To learn how to upgrade/migrate to Drupal 8, follow our full Drupal 8 Migration Guide.

Drupal 8 core provides support for Drupal-to-Drupal migrations. Since there is no direct upgrade path for Drupal 6 or 7 to 8, you should become familiar with the migration system in Drupal, as it will allow you to migrate your content from previous versions to Drupal 8.

In this post, which is aimed at advanced Drupal users, we will discuss how you can conduct a custom Drupal-to-Drupal to migration using core, and contributed modules, along with Drush.

If you have not yet read Mike Ryan's excellent blog post about the changes to the Migrate System in Drupal 8.1, I would highly recommend it.

Preparation

To be able to run custom Drupal-to-Drupal migrations in Drupal 8, you will need the following:

  • Drupal 8.1 (or greater) installed
  • A database backup from your Drupal 6 or 7 site, and optionally, your sites/default/files directory from your Drupal 6 or 7 site
  • A database backup of your freshly installed Drupal 8.1

Enable the following modules in your Drupal 8.1 site:

Core:

  • Migrate
  • Migrate Drupal

Contributed:

After enabling the required modules, add an additional database definition to your settings.php for your Drupal 8.1 site. See Drupal\migrate\Plugin\migrate\source\SqlBase

// Database entry for `drush migrate-upgrade --configure-only`
$databases['upgrade']['default'] = array (
  'database' => 'dbname',
  'username' => 'dbuser',
  'password' => 'dbpass',
  'prefix' => '',
  'host' => 'localhost',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);
// Database entry for `drush migrate-import --all`
$databases['migrate']['default'] = array (
  'database' => 'dbname',
  'username' => 'dbuser',
  'password' => 'dbpass',
  'prefix' => '',
  'host' => 'localhost',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);

Generate a migration

With these steps complete, it's time to generate a migration. Open up a terminal, and issue the following command, from your Drupal 8.1 root directory:

drush migrate-upgrade --configure-only

If you didn’t create an entry in your settings.php you can pass your database credentials for your Drupal 6 or 7 site, and optionally, the path to the files directory like so:

drush migrate-upgrade --configure-only --legacy-db-url=mysql://dbuser:dbpass@localhost/dbname --legacy-root=/path/to/sites/default/files

This command will generate migration configuration entities in the active configuration store, using migration plugins in Drupal core. Check out the tutorial, Configuration Data Storage, to learn more about the configuration system in Drupal 8.

Create a custom migration module

At this stage, we need to create a custom module for our migration, in your Drupal 8.1 site. You can do this manually, or using Drupal Console, like so:



drupal generate:module

// Welcome to the Drupal module generator
 
Enter the new module name:
> custom_migration
    
Enter the module machine name [custom_migration]:
>
    
Enter the module Path [/modules/custom]:
>
    
Enter module description [My Awesome Module]:
> A custom Drupal-to-Drupal migration
    
Enter package name [Custom]:
>
    
Enter Drupal Core version [8.x]:
>
    
Do you want to generate a .module file (yes/no) [no]:
> no
    
Define module as feature (yes/no) [no]:
> no
    
Do you want to add a composer.json file to your module (yes/no) [yes]:
> yes

Would you like to add module dependencies (yes/no) [no]:
 > yes

 Module dependencies separated by commas (i.e. context, panels):
 > migrate_drupal, migrate_plus
    
Do you confirm generation? (yes/no) [yes]:
> yes
    
Generated or updated files
Site path: /Users/willwh/Sites/drupal
1 - modules/custom/custom_migration/custom_migration.info.yml
2 - modules/custom/custom_migration/composer.json

Export site configuration

Create the directory custom_migration/config/install , which is where we will store our custom migration.

We can now export our site configuration, which will include our generated migration configuration entities. If you’re not familiar with the configuration system Drupal 8, you should check out our Configuration Management tutorials, specifically Manage Configuration with Command Line Tools.


drush config-export --destination=/tmp/migrate

Copy migration configuration to custom module

Next, we need to copy the migration configuration generated by drush migrate-upgrade --configure-only to our custom_migrate/config/install.

These files will be at /tmp/migrate and begin with migrate_plus.

Caution

Warning! Make sure you do not copy the default configuration group that is defined by migrate plus, i.e. migrate_plus.migration_group.default.yml.

Use the following command, and replace the last argument with the correct path to your custom module’s config/install location:

cp /tmp/migrate/migrate_plus.migration.* /tmp/migrate/migrate_plus.migration_group.migrate_*.yml /path/to/your/module/config/install/

Edit your module’s migrations

At this point, you can simply remove any of the migrations you don’t need, along with any dependencies on them. You can also now edit the migrations contained in your module to your liking.

For example, if you don’t want to migrate blocks from your previous site, you would delete the following files at custom_migration/config/install :

  • migrate_plus.migration.upgrade_block_content_body_field.yml
  • migrate_plus.migration.upgrade_block_content_type.yml
  • migrate_plus.migration.upgrade_d7_block.yml
  • migrate_plus.migration.upgrade_d7_custom_block.yml

Customize migrations with process plugins

Migrations may also be customized with process plugins.

Let’s say you’re migrating from a Drupal 7 site. If you wanted to map a node type from a previous Drupal version to a different node type in Drupal 8, you could accomplish this with the default_value process plugin.

For example, given this migration template:

  • migrate_plus.migration.upgrade_d7_node_blog_post.yml

In the process: section of the migration, take note of the following:

process:
  type: type
  name: name
  description: description
...

Instead of mapping the node type in Drupal 7 to one of the same name in Drupal 8, which, in my example would import the blog_post content from Drupal 7, to a content type of blog_post in Drupal 8, we can use the default_value plugin, and specify a node type of a different name.

In the process: key, change the values for plugin to default_value and value to the machine name of your desired node type.

process:
  type:
    plugin: default_value
    value: desired_node_type
  name: name
  description: description
  help: help
  title_label: title_label
  preview_mode: constants/preview
  display_submitted: display_submitted
  new_revision: options/revision
  create_body: create_body
  create_body_label: body_label

Run the customized migration

To run your migration, you must import your clean backup of Drupal 8.1, enable the required modules above, and your new custom_migration module.

You can check that you have everything set up correctly by running drush migrate-status. You should see a list of all of the migrations you have defined in your module.

A portion of my custom migration status looks like this:



Group: default                                  Status  Total  Imported  Unprocessed  Last imported
 upgrade_block_content_type                      Idle    1      0         1
 upgrade_d7_dblog_settings                       Idle    0      0         0
 upgrade_d7_image_settings                       Idle    0      0         0
 upgrade_d7_node_settings                        Idle    1      0         1
 upgrade_d7_search_settings                      Idle    1      0         1
 upgrade_d7_url_alias                            Idle    4953   0         4953
 upgrade_d7_user_flood                           Idle    0      0         0
 upgrade_d7_user_mail                            Idle    1      0         1
 upgrade_menu_settings                           Idle    0      0         0
 upgrade_search_page                             Idle    1      0         1
 upgrade_taxonomy_settings                       Idle    0      0         0
 upgrade_text_settings                           Idle    0      0         0

To execute your migration, run the following:

drush migrate-import --all

Stay tuned…

We will cover this process in much greater detail in our upcoming Migrate to Drupal 8 tutorials, but I hope this will help you get started in migrating to Drupal 8!

Update (May 6, 2016): Our Drupal 8 Migration Guide is underway. Check out the latest tutorials on Drupal-to-Drupal migrations here.

Comments

I have followed this tutorial to migrate a site from D7 to D8, the D7 site was a D6 I migrated several years ago, and I've been getting issues when running drush migrate-status and drush migrate-import --all that made me think I had artifacts from the D6 migration. For example:

SQLSTATE[42S02]: Base table or view not found: 1146 Table 'd2d_drupal7.book' doesn't exist: SELECT COUNT(*) AS expression
FROM
(SELECT 1 AS expression
FROM
{book} b
INNER JOIN {menu_links} ml ON b.mlid = ml.mlid) subquery; Array
(
)

SQLSTATE[42S02]: Base table or view not found: 1146 Table 'd2d_drupal7.filter_formats' doesn't exist: SELECT COUNT(*) AS expression
FROM
(SELECT 1 AS expression
FROM
{filter_formats} f) subquery; Array
(
)

Given this, I attempted to do a D2D migration from D7 to D7 in the hopes I'd get rid of the artifacts. After a lot of effort I got the site into a clean D7 installation. But as luck would have it, the same errors. So to be certain I ran the migration against a completely clean, nothing added D7 installation, just install.php with no modules turned on, and I still get the errors. I'm completely at a loss. Why is the D7 to D8 migration looking for deprecated D6 tables even on a clean virgin install of D7? Any insight would be very much appreciated.

I've tried to boil this down to as simple a migration as possible so I can actually witness something successful when migrating from D7 to D8. I have created 1 user and and 1 node with 1 custom text field. Having followed this tutorial, I can successfully import fields for instance, but not nodes, because nodes has a user dependency, and importing user fails and I am not able to track anything useful down as to why migrate-import does not like migrate_plus.migration.upgrade_d7_user.yml. This is what I get:

exception 'InvalidArgumentException' with message 'Passed variable is not an array or object, using empty array instead' in [error]
/home/drupal8/webroot/core/modules/migrate/src/Plugin/migrate/process/Flatten.php:51
Stack trace:
#0 /home/drupal8/webroo...

The two sites are as barebones as a drupal site can be. It just seems like this should be as simple as it gets.

Thank you,
Sean

I've not seen these errors before, but it's also been a few months since I've tried to run a migration of any sort.

My first suggestion here would be to make sure that you've got the most up-to-date version of the Migrate Plus module. And, that you're using the correct version for your minor version of Drupal 8. There's a list on the module page. https://www.drupal.org/project/migrate_plus - I would also make sure you're using the -dev versions of the code and not the "release" versions. It looks like a lot of changes have been made since the last release of either version of this module. And this whole system is still very much a work in progress.

After that, I would take a look at the issue queue for the migrate_plus module. https://www.drupal.org/project/issues/migrate_plus?categories=All

A little of quick searching says it might be related to this issue, which was just recently closed. https://www.drupal.org/node/2875405, or maybe this one https://www.drupal.org/node/2788139

Hopefully that helps get you going in the right direction.

Hi, William,

That was an awesome stuff on migrations.

Can I migrate images from Drupal6 (Custom CCK) to Drupal8?

I've tried with field.field.node.{ custom_cck }.field_image

it did not work.

Any help regarding this is much appreciated.

Thanks a Ton.

I followed the instruction but when i try to migrate files or images they dont work. Any idea on how to get this done. Thanks

Hi Doe,

In order to help you trouble shoot we'd need more information about the error messages you're seeing, or what more specifically isn't working.

Feel free to contact us via our support form and hopefully we can help get you on the right track.

Cheers,

Blake

I do everything as you say but all "drush migrate-import" does is importing everything from /tmp/migrate/ and not caring about my new module. It is enabled. Any ideas?

Hi Cristina,

I think you're on the right track. The first thing to do is definitely make sure your custom module is enabled. If things still aren't working after that you'll probably want to try clearing drush's cache (`drush cc drush`) as a next step.

Good luck!

Great clear post, thanks you. I wonder how can I use it for a D8 to D8 migration if at all. Any hints would be greatly appreciated.

At the moment Drupal 8 core doesn't contain migrate source plugins to extract data from Drupal 8. Meaning this process won't work for Drupal 8 to Drupal 8 migrations. You'll need to write your own source plugins, and/or find existing ones in contrib, to help teach the Migrate API how to get data out of an existing D8 site.

There's an issue here https://www.drupal.org/project/drupal/issues/2607524 to add Drupal 8 source plugins, though it doesn't appear to have gotten a whole lot of movement yet.

I face an error while making and upgrade using browser. It is "Source Database is Drupa8 version but Drupal7 is selected". I face this even when I provide wrong username and password and host.

please guide me about. Thanks !

Add new comment

Filtered HTML

  • Web page addresses and email addresses turn into links automatically.
  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <code class> <ul type> <ol start type> <li> <dl> <dt> <dd><h3 id> <p>
  • Lines and paragraphs break automatically.

About us

Drupalize.Me is the best resource for learning Drupal online. We have an extensive library covering multiple versions of Drupal and we are the most accurate and up-to-date Drupal resource. Learn more