Material stocks are an important part of the social metabolism. Owing to long service lifetimes of stocks, they not only shape resource flows during construction, but also during use, maintenance, and at the end of their useful lifetime. This makes them an important topic for sustainable development. In this work, a model of stocks and flows for nonmetallic minerals in residential buildings, roads, and railways in the EU25, from 2004 to 2009 is presented. The changing material composition of the stock is modeled using a typology of 72 residential buildings, four road and two railway types, throughout the EU25. This allows for estimating the amounts of materials in in-use stocks of residential buildings and transportation networks, as well as input and output flows. We compare the magnitude of material demands for expansion versus those for maintenance of existing stock. Then, recycling potentials are quantitatively explored by comparing the magnitude of estimated input, waste, and recycling flows from 2004 to 2009 and in a business-as-usual scenario for 2020. Thereby, we assess the potential impacts of the European Waste Framework Directive, which strives for a significant increase in recycling. We find that in the EU25, consisting of highly industrialized countries, a large share of material inputs are directed at maintaining existing stocks. Proper management of existing transportation networks and residential buildings is therefore crucial for the future size of flows of nonmetallic minerals.