All businesses need data to work. Whether it’s simply the customer’s name and address, or the details present in a wide range of products that they sell, or even how the stock behaves with each delivery to the consumers and the accounts change, every business ultimately base their decisions on good, reliable data.
Knowing this, we understood that we had to come up with the best way to export our customers’ entire database from their current system and move it into a robust ERP system that could handle the interactions within the dynamic dataset.
For our purpose, we eventually chose Frappe’s open-source ERP project, named ERPNext and based on their own Frappe framework.
No endeavour is without obstacles, and we, along with a lot of other ERPNext adopters, faced some interesting problems while trying to import large amounts of data records into the ERPNext system using the native Data Import Tool.
Freezes during Data Import
One of the first and most common issues that we faced during such imports was a complete halt in the process. The import bar wouldn’t budge ahead, nor would it display any clear progress status or show error messages.
After checking our console for errors, we discovered that the import was constantly facing time-outs. Doubling our timeout values didn’t prove to be useful. Strikingly, we faced no issues while previously testing out the import on a non-production instance of ERPNext with ‘Developer Mode’ active.
After consulting with the devs and other users on the official forum, we bumped up our gunicorn and nginx session timeout values, and tripled our initial number of gunicorn workers. By tailoring the suggested solutions for our system, we successfully managed to avoid import freezes on our production instance.
Even though we didn’t receive any fruitful solutions from the forum for our particular issues, they were quickly resolved not long after we created the threads on the forum and made the developers aware of their existence.
Not every software can be made without errors, but we realized that keeping in touch with the people running the system not only eventually helps us, but it also creates a channel for providing feedback and mutually improving the system for the rest of the users. Win-Win-Win.
There are many avenues that you can take to arrive at a particular state of imported data. However, to ensure that the data has been carefully and reliably regenerated in the ERPNext system will require certain best practices to be undertaken.
Use the command-line for large imports
When you’re importing data using a large data file (~5000 rows of records), prefer using ‘bench’ commands over the web UI for a more seamless process that isn’t plagued by constant timeouts and interruptions.
The bench command, ‘bench import-csv’ can be extremely useful for importing a large database.
Understand the type of data you’re importing
Not all data is the same, and this is particularly visible when you’re importing them into the system. Some data types (DocTypes in ERPNext) in the system rely on other data types to be already present for effective linking.
Ensure that they are already present before trying to import the dependent data types.
In the same vein, the maximum number of records that you can import in one ‘write’ process will also depend on the inter-linking of the data types in the system. For example, trying to import more than 5-10 companies at a time in the system can cause it to considerably slow down, since it’s dynamically and automatically setting up bank accounts, creating warehouses, etc.
Break down your import data into bite-sized pieces
Trying to import a large file may slow down or freeze your system and eventually waste valuable time.
Prefer splitting your data files into separate .csv files that hold 500-1000 rows of data (depending on the type of data) before importing them.
Rohan is a Junior Full-stack Python / HTML5 ERP Next developer in DigithinkIT, Inc.