How can you contact us

2By phone at +40 749 014 360

If you still have problems, please let us know, by sending an email to info@westweb-solutions.com Thank you!

OFFICE HOURS

Mon-Fri 9:00AM - 6:00AM
Sat - 9:00AM - 1:00PM
Sundays by appointment only!

Drupal 8 - How to use Migrate API

/ / Drupal 8

So, you need to import a large quantity of data. The problem is that that data can't be converted into a simple node.

Unlike previous Drupal 8 releases, this version has integrated into its core the Migrate module.

Let's say we have a Drupal structure similar to the one below

and a table source like this one

It's quite clear that we cannot import the data into simple nodes without using some preprocessing due to the defined relationships in our system (e.g. the "car brand" value is a node in our system, the "dealer" is a user entity).
This is were the Migrate module comes in really handy. These being said, let's get down to business!
 

1. Generate a new module that will define the source.

In the terminal, run the following command

drupal generate:module

add "migrate" as a dependency and proceed with the operation
Next, we need to generate the source plugin for our new module.

drupal generate:plugin:migrate:source


You should see something like this:

<?php

namespace Drupal\custom_import\Plugin\migrate\source;

use Drupal\migrate\Plugin\migrate\source\SqlBase;

/**
 * Provides a 'CustomImportTable' migrate source.
 *
 * @MigrateSource(
 *  id = "custom_import_table",
 *  source_module = "custom_import"
 * )
 */
class CustomImportTable extends SqlBase {

  /**
   * {@inheritdoc}
   */
  public function query() {
    return $this->select('custom_car', 'cars')
      ->fields('cars');
  }

  /**
   * {@inheritdoc}
   */
  public function fields() {
    $fields = [
      'id' => $this->t('ID'),
      'Name' => $this->t('Name'),
      'Dealer Name' => $this->t('Dealer Name'),
      'Dealer Phone' => $this->t('Dealer Phone'),
      'Model' => $this->t('Model'),
      'Brand' => $this->t('Brand'),
      'Price' => $this->t('Price'),
    ];

    return $fields;
  }

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    return [
      'id' => [
        'type' => 'text',
        'alias' => 'cars',
      ],
    ];
  }

}

 

2. Install the migrate_plus module and add the configuration ymls for the import plugin source.

Now, let's create the two yml files that will define the migration group and the migration configuration for our previously defined plugin. Their location is custom_import/config/install.

2.1. Migration Group - migrate_plus.migration_group.custom.yml

# A "migration group" is - surprise! - a group of migrations. It is used to
# group migrations for display by our tools, and to perform operations on a
# specific set of migrations. It can also be used to hold any configuration
# common to those migrations, so it doesn't have to be duplicated in each one.

# The machine name of the group, by which it is referenced in individual
# migrations.
id: custom

# A human-friendly label for the group.
label: Custom Imports

# More information about the group.
description: Custom import.

# Short description of the type of source, e.g. "Drupal 6" or "WordPress".
source_type: Custom sources

# Here we add any default configuration settings to be shared among all
# migrations in the group. For this example, the source tables are in the
# Drupal (default) database, but usually if your source data is in a
# database it will be external.
shared_configuration:
  # Specifying 'source' here means that this configuration will be merged into
  # the 'source' configuration of each migration.
  source:
    # A better practice for real-world migrations would be to add a database
    # connection to your external database in settings.php and reference its
    # key here.
    key: default

The result:

2.2. Migration - migrate_plus.migration.custom_import_table.yml

This is where we'll do the mapping (set the new node type, set the title, set other default values and which fields should be used for mapping)

# Migration configuration for car content.
id: custom_import_table
label: Custom Cars
migration_group: custom
source:
  plugin: custom_import_table
  key: default
destination:
  plugin: entity:node
  bundle: 'car'
process:
  type:
    plugin: default_value
    default_value: 'car'
  title: name
  uid:
    plugin: default_value
    default_value: 1
  status:
    plugin: default_value
    default_value: 1
  sticky:
    plugin: default_value
    default_value: 0
  field_price: price

The result:

At this point, we can import the content; all the new nodes will have information about the name and the price.

 

3. Generate the processors for the entity reference fields.

3.1. Taxonomy term processor

The car model source equivalent in our system is the "Car Type" taxonomy. We'll have to process the string and either create a taxonomy term for the given value or use an existing one.
Let's update the /config/install/migrate_plus.migration.custom_import_table.yml by adding the car type processor.

  field_car_type:
    plugin: car_type
    source:
      - Model
After you update the migration ymls, you may have to reinstall the module in order to rebuild the migration configurations.

In the /src/Plugin/migrate/process, we'll have to create the processor for the Car Type. The content of the CarType.php file is below:

<?php

/**
 * @file
 * Contains \Drupal\custom_import\Plugin\migrate\process\CarType.
 */

namespace Drupal\custom_import\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;
use Drupal\taxonomy\Entity\Term;


/**
 *
 * @MigrateProcessPlugin(
 *   id = "car_type"
 * )
 */
class CarType extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $row_data = $row->getSource();

    if ($row_data['Model']) {
      // Load term by name.
      $terms = \Drupal::entityTypeManager()
        ->getStorage('taxonomy_term')
        ->loadByProperties([
          'name' => $row_data['Model'],
          'vid' => 'car_type'
        ]);

      if (! ($terms && $car_term = reset($terms))) {
        $car_term = Term::create([
          'name' => $row_data['Model'],
          'vid' => 'car_type',
        ]);
        $car_term->save();
      }

      $value = [
        'target_id' => $car_term->id(),
      ];
    }

    return $value;
  }
}

 

3.2. Node processor

The brand will be saved as a new entity and its id will be added as a reference to the car.
Let's update the /config/install/migrate_plus.migration.custom_import_table.yml by adding the brand processor.

  field_car_brand:
    plugin: car_brand
    source:
      - Brand
After you update the migration ymls, you may have to reinstall the module in order to rebuild the migration configurations.

In the /src/Plugin/migrate/process, we'll have to create the processor for the Brand. The content of the CarBrand.php file is below:
 

<?php

/**
 * @file
 * Contains \Drupal\custom_import\Plugin\migrate\process\CarBrand.
 */

namespace Drupal\custom_import\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;
use Drupal\node\Entity\Node;


/**
 *
 * @MigrateProcessPlugin(
 *   id = "car_brand"
 * )
 */
class CarBrand extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $row_data = $row->getSource();

    // Load term by name.
    $nodes = \Drupal::entityTypeManager()
      ->getStorage('node')
      ->loadByProperties([
        'title' => $row_data['Brand'],
        'type' => 'car_brand'
      ]);

    if (! ($nodes && $car_brand = reset($nodes))) {
      $car_brand = [
        'nid' => NULL,
        'uid' => 1,
        'status' => Node::PUBLISHED,
        'title' => $row_data['Brand'],
        'type' => 'car_brand',
      ];
      $car_brand = Node::create($car_brand);
      $car_brand->save();
    }

    $value = [
      'target_id' => $car_brand->id(),
      'revision_id' => $car_brand->getRevisionId()
    ];

    return $value;
  }

}

3.3. User processor

The dealer will be saved as a new user and its id will be added as a reference to the car.
Let's update the /config/install/migrate_plus.migration.custom_import_table.yml by adding the dealer processor.

  field_dealer:
    plugin: dealer
    source:
      - Dealer Name
      - Dealer Phone
After you update the migration ymls, you may have to reinstall the module in order to rebuild the migration configurations.

In the /src/Plugin/migrate/process, we'll have to create the processor for the Dealer. The content of the Dealer.php file is below:

<?php

/**
 * @file
 * Contains \Drupal\custom_import\Plugin\migrate\process\Dealer.
 */

namespace Drupal\custom_import\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;
use Drupal\User\Entity\User;
use Drupal\paragraphs\Entity\Paragraph;

/**
 *
 * @MigrateProcessPlugin(
 *   id = "dealer"
 * )
 */
class Dealer extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $row_data = $row->getSource();

    // Load term by name.
    $users = \Drupal::entityTypeManager()
      ->getStorage('user')
      ->loadByProperties([
        'name' => $row_data['Dealer Name'],
      ]);

    if (! ($users && $dealer = reset($users))) {
      $dealer = [
        'status' => 1,
        'mail' => 'mytestdeveloperaddress+' . str_replace(' ', '-', $row_data['Dealer Name']) . '@gmail.com',
        'name' => $row_data['Dealer Name'],
        'role' => ['car_dealer'],
        'roles' => ['car_dealer'],
        'field_phone' => $row_data['Dealer Phone'],
      ];
      $dealer = User::create($dealer);
      $dealer->save();
    }

    $value = [
      'target_id' => $dealer->id(),
    ];

    return $value;
  }

}

Bonus: you may also need to add more preprocessing to your data. You may do that in your source plugin /src/Plugin/migrate/source/CustomImportTable.php in the prepareRow method.

  /**
   * {@inheritdoc}
   */
  public function prepareRow(Row $row) {
    $row_source = $row->getSource();

    foreach ($row_source as $field_name => &$value) {
      $value = trim($value);
      if ($value == 'NULL') {
        $value = NULL;
      }
      $row->setSourceProperty($field_name, $value);
    }
  }

Just make sure to include the Row class at the beginning of the file.

use Drupal\migrate\Row

Sign in your account to have access to different features

Forgot your details?

TOP