Indoor climate control is a major energy demand everywhere. Design and operation of buildings and HVAC systems crucially depend on climate data and real-time meteorological conditions. Energy-efficient buildings also contribute to reduce air pollution and climate change in urban areas as well as regionally and globally. However, the effects of climate and weather on building energy management are still largely overlooked in practice. A main reason for this is the lack of tools for translating meteorological conditions into energy requirements. The combined impact of temperature, solar irradiation, wind and humidity on the energy balance of a building depends on the building itself, i.e., its design, orientation, HVAC system, mode of operation, maintenance, etc. The paper discusses different approaches to model this complex interplay and associated problems at the design as well as in the operation stages. Recent developments in Sweden are reported, including applications to urban planning, building design and real-time operation of buildings and energy systems. The impact of solar irradiation and wind, in addition to that of temperature, is demonstrated. Further, the paper discusses the significance of local site condition versus building characteristics.