Large product catalogs, in Magento or any other platform…can be a pain. Creating products, categories, and attributes manually is often not an option. For most of our stores with very large product catalogs, years might not be enough to enter them manually. Bulk updates of existing products can also take huge amounts of time and effort. For reliable and efficient product data import, it’s essential to use an smooth and accurate tool capable of automating Magento data import. To learn more, read our guide on Magento Speed & Performance.
Fortunately, Magento 2 has a large selection of import tools. Some are extensions that require you to update CSVs, but these can be just as time consuming as doing it manually for you to prep all the data yourself. Other tools are automated 3rd party tools and can be pricey.
We usually recommend a different path to success. Magento 2 Core comes with a native tool that does all the work for you: the Magento 2 REST API (an acronym that stands for: Representational State Transfer Application Programming Interface. Whew!).
We can take any data source (CSV, database, another website, another Magento website) and create a script to format and send Magento 2 data in a format that handles creating new and updated products for you.
With this method, you can schedule regular data updates without you having to lift a finger: Magento can request your data source, generate the correct formatting for your products, and initiate the product import and updates automatically, as often as you would like.
Problem: Import a large product catalog to Magento all at once
Managing an e-commerce catalog is generally a lot of work, but what if your catalog has thousands of products? Anything manual is completely out the window.
Magento 2 has a CSV import system, but with thousands of products, it will take so much time for the server to import and iterate over your CSV rows that it will ultimately just run out of resources, failing to complete. Even splitting the CSV into chunks will take way too much time and effort to deal with in a way that won’t completely occupy all of your free time.
Enter the REST API: Magento 2’s way of communicating with the outside world. You can leverage the power of the REST API to handle all kinds of data import challenges.
What is the REST API?
The Magento 2 REST API framework provides developers services that can communicate with Magento 2. Some features include:
- You can safely authenticate services to send/receive data from Magento 2
- You can ensure proper permissions are honored depending on the data you are working with
- 3rd Party Services can be integrated with just a few lines of code in a configuration file
- You can easily work with database models like customers, addresses, orders, etc
- Magento provides APIs to deal with multiple services at once (for example, uploading products and categories at the same time)
We can develop scripting to automate any data source to work with these Magento APIs. Say your data source is a CMS, like WordPress or Sharepoint. We can create a script to connect to this data source and pipe it to Magento on a regular schedule, ensuring your changes will always migrate to Magento as frequently as possible.
What’s a typical product import look like?
We will take your data and format it in a JSON (JavaScript Object Notation) format, which is the most common way to send REST API’s data. So an example of a product import in this format would be:
Request:
{
"product": {
"sku": "A19458-SKU",
"name": "Your Killer New Product",
"price": 30.00,
"status": 1,
"type_id": "simple",
"attribute_set_id":4,
"weight": 3
}
}
Response:
{
"id": 999999,
"sku": "A19458-SKU",
"name": "Your Killer New Product",
"attribute_set_id": 4,
"price": 30.00,
"status": 1,
"visibility": 4,
"type_id": "simple",
"created_at": "2017-07-20 22:31:27",
"updated_at": "2022-06-07 21:21:13",
"weight": 3,
"extension_attributes": {
"website_ids": [
1,
2,
3
],
"category_links": [
{
"position": 1,
"category_id": "3"
},
{
"position": 1,
"category_id": "255"
}
],
"product_links": [],
"options": [],
"media_gallery_entries": [],
"tier_prices": [],
"custom_attributes": [
{
"attribute_code": "size",
"value": "Large"
}
]
}
}Magento can loop through thousands of products this way, matching data by SKU, or if no SKU exists, generating the product for you. It will import everything (i.e. all product attributes), and can be imported to one or all of several stores within the primary Magento store.
As an added benefit, when you use the Magento2 API to insert a product, you will receive a response that will tell you whether the product was successfully updated, and if it wasn’t, it will tell you what the problem was. This can be invaluable and save hours of troubleshooting time for a simple error.
CRON Scheduling
The scheduling of this automated import process will be handled by CRON (stands for: Command Run On). All web servers can handle scheduled requests using CRON. We would typically schedule your import as frequently as possible, based on the amount of products in each request, and how long we expect Magento 2 to work on importing the data.
With the REST API, you can import hundreds of products per second.
If you find that you would appreciate some assistance in importing hundreds of thousands of products or just a few catalog updates, reach out to us. We can help make the process painless, and we can examine the best way to automate your data so you can spend your valuable time in more meaningful ways instead of being bogged down by spreadsheets.
