Community Portal

Better deal with large shapefiles

Created on Tuesday 17 November 2020, 07:45

Back to task list
  • ID
  • Project
    Metabolism of Cities Data Hub
  • Status
    In Progress
  • Priority
  • Type
    Programming work
  • Assigned to
    Paul Hoekman
  • Subscribers
    Paul Hoekman

You are not logged in

Log In Register

Please join us and let's build things, together!


We're now getting many more shapefiles loaded into the system, which is great. However, some are 10... 20.... 30... 50 MB files! This becomes crazy large to load on a single map. Also, pulling in all that geometry takes time in the back-end, so in general we must just better figure out what to do with this. Also, I've seen some crazy detailed files about very specific places (the type of paint used throughout the port of Barcelona, stuff like that). So we must also question what we need and what we don't need. The reference spaces table is growing by leaps and bounds which again is great but 100k items just to describe this single port seems excessive. So let's figure out how we best approach this.

Discussion and updates

New task was created

Task was assigned to Paul Hoekman

Status change: Open → In Progress

Some first changes include:

  • When a user wants to process a huge shapefile (> 1000 items), a warning pops up, see attached

  • When a map is opened that is already processed but that contains > 500 items, then the system will only show the first 500 items on the map

  • When a map is opened that is more than 5 / 10 / 20 MB, then the system will use a simplification script that makes lines in the polygon less detailed but much smaller (the level of simplification increases depending on the file size). The user is still given an option to open the non-simplified map, but it can take ages so the user is warned.