A new caching system has been set up. Just to keep the technical functioning documented:
- The Django caching system has been enabled
- We specifically cache the json objects that are needed as data input for our charts. These objects are time-consuming to create because we loop over all the data to format it in the right way. However, for every dataset there is no need to keep doing this over and over, so it's ideal to cache.
- We cache indefinitely. However, we keep track of what is cached because when a dataset is updated we need to remove it and re-generate the cache.
- In the meta_data of the library item we store which cache objects have been created so far
- In principle, the cache is stored the first time a chart is displayed (the system sees it does not exist and then stores it in cache)
- However, for some charts the loading time > 30 sec and a server time-out happens, so nothing is ever stored. For those datasets there is an admin-button in the left-hand menu (the total number of buttons got too wild so I hid them away under an "admin options" button to keep it all a bit cleaner). When clicking this button, the system will schedule the cache creation server-side, where there is no problem with the time-out. This is done through a cronjobs.
- There are certain "sliced" data visualizations (e.g. when looking at a spatial subset of the data). These are separate json objects. In order to distinguish between these, there is a cache-key generated that includes all GET parameters (because we pass various conditioning parameters through the URL). Each cache object is saved with a unique cache key that varies even if it's for the same dataset, so that we can have multiple cache objects for individual datasets.
Let's see how this performs over the coming weeks and months, but I can at least report that 20 second generation time is down to less than 1 second for those that I have checked, so looking much better!