Shopify app Excelify now offers the ability to create a Batch import job that will monitor your FTP/SFTP folder and import any files automatically that you put in this directory.
This way you can set up a monitoring job for your server directly that continuously watches your folder. No need to manually import each file, just put it in your FTP/SFTP folder!
Quick summary on how to set up an FTP/SFTP server directory monitoring job
- Set up folders in your FTP/SFTP server.
- Construct FTP/SFTP directory link with credentials.
- Create an initial batch job by pasting link to “Upload from URL”.
- Inspect your import and set up Options:
- Set up rules to move your import file at the start of import
- Set up where to move Results file (optional)
- Set up how often the Excelify should import from your directory
- Press the “Import” button to start your Batch import.
- Follow the progress of the import and enjoy!
Once you have gone through these steps, the app will create a “child” job for each file in a directory that you imported.
You can track all your file import progress right from this Batch job or you can click on the individual job to see more details about it.
At the start of the directory import, the app will already create the next import if you have set up an option to repeat.
Do note – importing whole directory, not a specific file is only available for Excelify Big or Enterprise plan users.
Let’s go through each step to import your server’s directory.
1. Set up folders in your FTP/SFTP server
First, let’s understand how the batch monitoring process will work on your server.
In essence, you would set up a repeated Batch import job that will run at each specified time interval, e.g. every 5 minutes. It will check the specified directory on your server and import any CSV, XLSX or ZIP files that are there.
To ensure that after 5 minutes the same file is not imported again, we can set up the Import Options to move imported files to a different folder, or to delete the file when it’s imported.
Additionally, the app also returns Import Results file for each file that you import, so optionally, you can set up to upload that to your server too – in the Import Options.
In this tutorial, we will be using ExaVault.com as a server host, but you can use any software or host for your FTP/SFTP server that allows you to create and edit folders and files.
I will set up a separate folder in my server for Excelify imports, called “Excelify-ImportExport”, this will work as my folder for all things Excelify.
In the Excelify-ImportExport folder, let’s a folder for each status – “Inbox”, “In Progress”, “Finished”, “Failed, “Cancelled” and “Results”. Those are just an example – you can create whatever folders you want, and in whatever path you need to have them.
- Inbox – This will be a directory that the app checks and imports any files from.
- In Progress – Here the file will get moved when the Excelify starts to import it.
- Finished – In this directory, the Excelify will move file once it’s done importing (regardless if there were any failed items in that file).
- Failed – Here the file will get moved if the whole file crashes/fails in the app. For example, if Excelify was not able to recognize its template.
- Cancelled – If the job is cancelled for any reason, then it will get moved here.
- Results – In this directory, the Excelify will upload the Import Results file once each file import is finished.
2. Construct FTP/SFTP directory link with credentials
You will need to construct your server link. This needs to include your username, password, server and path to the directory.
For FTP server the link would be constructed in the following template:
For SFTP server the would be constructed almost the same, as per the following template:
* The port needs to be specified only if it’s not the default port.
Let’s assume that you have following credentials for your SFTP server:
- SFTP server address is: my-data-server.com
- SFTP user name is: master
- SFTP server password is: secure123
- Path to the directory: /Excelify-ImportExport/Inbox/
The resulting URL would be:
3. Create “Upload From URL”
Once we have our server folders set up and URL at hand, we can go into the Excelify app and create our batch import job from the directory.
In the Home page of the app, in the “Import” section you have a field where you can fill URL that you wish to import from.
Paste your constructed URL in this field and press “Upload From URL”. The app will take you to a new page where you will be able to inspect and set up your repeated import.
4. Inspect and set up your Batch import
In the new Batch import page, first, you will see basic information about the import job.
You can see such things as format, plan, created dates and more.
The Format of the import job should be “Batch”.
If your folder already contains any CSV or XLSX files then you will also see those listed here and they will get imported when you accept to import the job.
First, you see the total amount of items and an approximate estimate of how long it would take. Below that, you see each file that will get imported.
Inspect how many items each file contains and it imported entity (Products, Customers, etc.) matches what you are importing.
Further below you will see the Options section. Let’s edit Options to set up the continuous batch import that will monitor our import directory and will repeat importing all files that appear there:
Set up Options to move your import file by folders
First, in the Options, we need to set up the option “What to do with remote SFTP file when importing?”
Here you can choose one of three settings:
- Do nothing – nothing will happen to your file in your server it will just get import and stay there.
- Delete that file on SFTP server – the app will delete each file it imports.
- Move file to directory by Import status – moves the file to a different directory based on its current status within this batch job.
Lets select “Move file to directory by Import status” so that the app moves our file automatically.
This will show us 4 options that we can set:
- In Progress – the app will move the file to a specific directory once the app starts to import it.
- Finished – the file will get moved to the specific folder once it finishes importing (the file will be moved here even if there are some failed items inside the import, because in principle that import was finished).
- Failed – move the file if the whole job file fails to import (in most cases, when the template format is not recognized at all).
- Cancelled – move the file if that import job was cancelled.
We created a separate folder for each stage of the file import so the file gets moved around. We can now set those directories in the app.
In our example setup it would be like this:
- In Progress – “/Excelify-ImportExport/In Progress”
- Finished – “/Excelify-ImportExport/Inbox”
- Failed – “/Excelify-ImportExport/Failed”
- Cancelled – “/Excelify-ImportExport/Cancelled”
Note that you need to include the full directory path to that folder on your server. That allows you to put those directories on any path. And those folders need to exist, so you need to create them first.
Set up where to move Results file (optional)
Now further below you see a lot of checkboxes, you can leave all these as they are, for default importing it should be fine.
Check here for more information about all the Options.
If you want Results file to be uploaded to your server, find and fill the option “Upload Results file to:“.
Each job will save and return the Import Results file in the app, but you can also upload it automatically to your server.
Click on the dropdown to select if you wish for it to upload to FTP or SFTP server.
Once selected, fill the URL with username and password that leads to the server and directory path where you wish for this file to get saved.
In my example, this would be:
Set up how often the Excelify should import from your directory
At the very end of the Options section, you can set up scheduled and repeated import.
In this case, we would not set up “Schedule on:” option as we want our import to start now.
Check the box for “Repeat every:” option and set the time interval how often do you want Excelify to repeat the Batch import from your server directory. The most frequent is every 1 minute, but note that it means it will generate a new Batch import each minute and will connect to your FTP/SFTP server to check if there are any files to import, which might be exactly what you need.
5. Start your initial Batch import
The batch import that monitors your server directory every 1 minute should be now set up all correctly.
Take one more quick look over your setup and press the “Import” button to start the job.
If your directory already contained any files then you can follow the job as they are being imported.
You can also click on each of the sub-jobs to open them and see more details about each of them.
If the folder did not contain any files then the job will just finish.
At the moment when you start this import, the app will already create the next one that should start after 1 minute.
If the current one will not yet be finished after 1 minute, then the next job will go automatically in the queue and start once currently running job finishes.
This will ensure that there are no multiple jobs running over each other and make a steady stream of imports.
You can go back to the Excelify app Home and go into “All Jobs” page where you should see the repeated batch import happening every 1 minute where the app checks your directory.
You will see the summary information about each job. Click on the Batch job to see more details about it.
To stop the repeated importing – just cancel that Scheduled job.
Good to know!
- If your directory contains multiple files, the app will import them in alphabetical order.
- In the batch job you can cancel each individual file import if there is a specific file that you do not wish to import, this can also be done while the files are importing.
- In Excelify app Settings, you can set to receive notification emails about specific job statuses. Decide – in which cases you want to receive an e-mail, and when not, to not overload your mailbox.