Community Portal

Process dataset

Created on Thursday 6 August 2020, 15:37

Back to task list

You are not logged in

Log In Register

Please join us and let's build things, together!

Discussion and updates


This task was created by the system


Task was assigned to Aristide Athanassiadis and status was changed: Open → In Progress


Hey Paul, just processing also the embodied requirements of the material stock of Melbourne and I get the error that we do not have GJ yet. Should I multiply/divide my current figures or do you want to add GJ?


Yes! Something I can help with! :) Here is an overview of all the units that we have and conversion tables to also better see what our primary units are. You can add a unit yourself, by clicking ADD UNIT at the bottom of the page or using this link. Curious to see if the system then recognises it, since the unit was added afterwards. Do share what happens.
Luckily, I had made a note of Paul's original input on that ;)


Haha, you're on top of things Carolin! I had seen the overview of all units indeed.
The thing is that Paul has a warning comment once you get there :) So I'm not sure whether I should touch something or not. Will report back though after that.


Ah yes thanks for asking Aris. Please go ahead and add the GJ unit. After that, you can reprocess your dataset.


Got it thanks to you both!


Ok added the unit and tried to process the dataset and get the same error as before (Hey Paul, thanks for letting me know. I've tried to process it and every time I assign the ref space (Embodied ...) and click on save & next, I see there is some activity (meaning this wheel instead of the favicon in chrome), I then see the screen MoC is updating and then it goes back to the same processing screen this time with none of the reference space selected.
Any pointers?
).
Should I would anything more or you are taking it from here Paul?


Yeah this must be the same issue as the other dataset. It is too large for processing and will require some intervention from my side. I'll look into it.


Hey Paul, did you get the chance by any chance to have a look into this?


Not yet Aris, been busy with some other stuff. Maybe tomorrow if not next week.


Yeah coolio Paul!


Hey Aris,
Okay I've now sat down for this. The system has been modified so that large datasets (> 1000 items) are not processed on the fly, but are instead slotted into our processing queue. They are then processed server-wide, every 7 hours. So, at this point you should be able to complete the processing steps and then let's see what happens once this has run on the server!


Status change: In Progress → Completed


Thanks Paul, I gave it a go. Let's see what happens in 7 hours!!


Great stuff Aris, let's see!!


Hey I think it worked!!! Just one thing, for the environmental requirements dataset it does not recognise the kl unit. I've checked in the STAF website and the unit is kL. Is capitalisation considered in the units? If so we should (I can do it if ok) change the unit to kl. All units that are not referring to a person should be smaller lowercase.

Should I check with Guus about continuing with Material Stocks


Great Aris!! And yes capitalization matters.

Please do check in with Guus about material stocks!


Status change: Completed → Completed


Ok great I changed the unit to kl in STAF and it now works! I hope I did not mess up any other already processed dataset.


Oh oh.... I thought you meant changing it to "kl" in your dataset! Can you please change it back to kL in STAF?? Otherwise indeed other datasets will be affected. Your dataset will remain processed so that is good, but if you want to reprocess it in the future you would have to fix the capitalization in your spreadsheet.


Can you identify by any chance which datasets have already used this unit? Ideally I would change these instead and keep the correct unit spelling in STAF.


What is the source for the correct spelling? I have always used kL and was under the impression this was correct (on Wikipedia it says they are both allowed...).


Here is the source

> The first letter of symbols for units derived from the name of a person is written in upper case; otherwise, they are written in lower case. E.g., the unit of pressure is named after Blaise Pascal, so its symbol is written "Pa", but the symbol for mole is written "mol". Thus, "T" is the symbol for tesla, a measure of magnetic field strength, and "t" the symbol for tonne, a measure of mass. Since 1979, the litre may exceptionally be written using either an uppercase "L" or a lowercase "l", a decision prompted by the similarity of the lowercase letter "l" to the numeral "1", especially with certain typefaces or English-style handwriting. The American NIST recommends that within the United States "L" be used rather than "l".

Ok apparently since 1979 it is also allowed to use kL (exceptionally, hehe). Hmm should we keep kL then and I change the dataset from Melbourne.


OK yes please, would be great. Thanks!


Status change: Completed → Open


Status change: Open → Completed


Hey Aris,

You will soon chat with Guus and we found that there are a number of data points missing. In other words, there is a building in the shapefile but no material data in the dataset. Maybe this is intentional. However, if you end up reloading the dataset, can I suggest that you round the values? Currently there are 14 decimals, which conveys an implied accuracy at the picogram level (this is one trillionth of a gram). Likely, your dataset is not at this accuracy level ;-)
I recommend simply rounding to the nearest kg. Also makes it all much more readable.