What’s the Shopify maximum import limit? (Updated on 15 Dec, 2017)

Searching For Maximum - 100'000 - progress - 4 hours remaining

With this exercise our goal is to find the Shopify maximum points. (Updated on Dec 15, 2017 with new speed improvements)

We will start with basic thing – by importing 100k Products, and move up from there.

Day 1: 100’000 Products Import

Before we started, it was estimating 2d 7h min to complete, which would be the time of single-stream import.

But we will have 10 streams importing in parallel, so let’s see where this takes us.

Shopify maximum import speed

Few minutes later the estimated time dropped to just 7h 9 min, because all the 10 parallel imports are working like crazy!
We are moving at the speed of 0.264 seconds per Product, which is basically 3.79 Products per second, or 227 Products per minute, or 13k Products per hour.

It’s done! Our Shopify store now has 100’691 Products!

Searching For Maximum - 100'000 - done

 

What next?

You make a guess! ūüėČ

OK, look! Pressing “Import” button now…

Big Import of Collections, Customers, Discounts - before start

Alright, the weekend has passed, and I admit, we had some bumps – we had wrongly formed Discounts import table which was failing. So we decided to cancel the import, fixed the table, and imported again. We want the “finding maximums” kind of results here.

Here they are:

As you can see, the first kind of maximum is found. That is – we have hit the Shopify limit of 5000 Smart Collections.

Other than that, 10k+ Custom Collections, 100k+ Customers, 20k+ Discounts. Not bad at all, Shopify! Very not bad at all!

 

Should we continue?

Off course, we should, and we will…

Tune into our Twitter account @excelify_io for further tests and updates.

Metafields import

Speed:¬†Metafields import speed doesn’t depend on Metafield count anymore. Even for 100 Metafields per item, it will import as quickly as with no Metafields.

Delete: You can delete Metafields for items just by clearing values in that Metafield column, and importing.

Set type:¬†Metafield can have “string” or “integer” type, which is now auto-detected from Excel cell type, and set correctly when you do the import.

Read more: Metafields template documentation

How to maximize Shopify bulk import speed?

Shopify import speed

There are different ways how you can gain maximum import speed when loading data to your Shopify store. Some are faster than others.

I will describe two basic import scenarios – one for importing with the goal to create new items, the other – with the goal to update existing items.

By “items” I mean: Products, Collections, Customers, Discounts, and so on.

While Shopify CSV file import is also quite fast, the following instructions apply to¬†“Bulk Import Export Update with Excel” Shopify app and its Excel file format for importing and exporting data.

Overall import logic

A bit of background logic, how importing to Shopify happens.

At first, an item is identified by “ID” column value. ID is the Shopify internal identifier for an item. You can’t assign them by yourself – you can only get them out by exporting existing items.

If the ID column value is empty or such ID doesn‚Äôt exist – then it tries to find the existing item by “Handle” value. In Discounts case the handle is the “Code” column value; in Customers case that is an “Email”, but for the sake of common terminology – app will refer to them as “Handle” in all those cases.

If it finds existing item by ID or by Handle, then it will update that item by changing only fields you provided in the import, and leaving other fields as they were. If it will not find such item, then it will create New.

So if you really want the fastest possible import speed, here’s what you do in each scenario.

Fastest import of New items

Keep the ID column empty, or remove it entirely from the import file.

Then it will not even try to find existing by ID, it will only check if item with such Handle exists – that is needed to determine if it needs to create New or do the Update of existing.

If your store is relatively small, import speed by Handle will be the same as by ID.

Fastest Update of existing items

First – help the app to identify existing items quickly

Shopify import upadte by ID

Have the ID column with ID values from the export file.

If you have ID for each item, and if item with such ID actually exists, then it will be the fastest way to update them – it will do one single “push”, and that’s it.

You don’t have to have ID – you can have just the Handle. Then it will find by Handle – it will be roughly two times slower, but still ok.

You will have the slowest speed if you have IDs which don’t exist. Because Then it will try to update by ID, which it will not find, then it will need to search by Handle, and then do the update. So it will be three times slower.

So – whenever you can, update by ID, the correct one.

Second – import only fields you need to update

Shopify import speed - delete unneeded columns

When you import your data, whatever item columns you import, all of them be sent to Shopify. So to improve the speed, import only those columns, which you really need to update, and avoid importing columns, which you are sure, are already correct.

Especially big difference is made with uploading images. Every time you import an image, Shopify will re-download that image. That adds at least one second to each item import, which might not seem a big deal, but it accumulates big time – imagine if you are importing several thousands of items. So, if you know you have your images already in place, just remove image columns from the import file.

Similar principle is about “related” details for each item. For example, do you need to update Product Variant data, when you are updating the Product Tags? Probably not. If you need to just update Product Tags, then you just remove all the Variant columns (and rows), and it will not have to touch Variants at all, again, gaining a bit of speed per item. Similarly, if you don’t need to update related Products for Manual Collections (Custom Collections), just remove those columns (and rows) form import file.

The same also with Metafields.

Pre-loading

In certain cases, the app will “pre-load” existing items from your store, to make the searching existing items faster. That’s because pre-loading is 250 times faster than identifying items one by one, so you can imagine the impact it has on overall import speed.

If you start the import and it seems like it’s not doing anything, give it a minute or two, because it decided that in total it will be quicker to pre-load your existing items, and then do the update on them.

Don’t check if items already exist

If you are absolutely sure that your import is just creating new items, and they don’t exist in Shopify, you can force the app not to load any of existing items by any of method. Just be aware that if items do exist, then there might be failed imports, errors, duplicates, or any other unexpected updates if those items exist.

Force not to check if items already exist in Shopify store

 

Which scenario was chosen?

You can see, how the existing item was identified, when you look in the Import Results Excel file (which you can download after import has finished).

In the “Import Comment” column you will see the following comments (in the order starting from fastest):

Update: Pushed by ID If you see this, it means you did the best job in helping the app to identify the existing item.
Update:¬†Found by ID (pre-loaded) Here the app searched for the existing item by ID in the pre-loaded items list, because it can’t push it, it needs additional data about the item to be able to do the update. You can improve this by removing columns, which you don’t need to update.
Update: Found by ID The same reason as for pre-loaded, just app decided that in this case searching items one by one is faster than pre-loading all existing.
Update:¬†Found by Handle (pre-loaded) Either you don’t have the ID, or the item with such ID doesn’t exist in this store. But it exists with the same Handle (or Code in case of Discounts, or Email in case of Customers). Also, app decided that pre-loading items will be the fastest way to find them.
Found by Handle The same reason as for pre-loaded, just app decided that in this case searching items one by one is faster than pre-loading all existing.
New The new item was created – meaning, that it didn’t find any existing item by ID or by Handle.

Summary import speed tips

  • Identify items by correct ID.
  • Remove columns and rows form import file which you don’t need to update.

What to do next?

  1. Review the “How it works?” tutorial.
  2. Install the “Bulk Import Export Update with Excel” app if you haven’t already.
  3. Upgrade to Big Plan or Enterprise Plan to give yourself to enjoy all the other benefits that come with them.