Software

VEDA (VErsatile Data Analyst)

VEDA means “Knowledge” in Sanskrit. It is a software tool to convert modeler’s knowledge into input for models, and output from models into knowledge. Veda2.0 is a data handling system for The Integrated MARKAL-EFOM System (TIMES) – a bottom-up optimization model for energy-environment systems. It uses C#.NET for UI and PostgreSQL as the backend. Veda is based on a modular approach that organizes the model input data, and results, into an integrated database. Information is visible via tabular browsing (data cubes) and network diagrams. It is used to develop and manage model runs and to analyse model results.

VEDA is under continuous development, driven by a very strong desire to keep increasing the efficiency and transparency of managing input (and output) of data-intensive models. The vision statement of VEDA applications is to take the drudgery out of modelling. The developer being its first user is a special feature of VEDA. Intimate domain knowledge has obvious advantages for software development.

Veda is a proprietary commercial software designed and developed by KanORS-EMR. It has been supported by ETSAP since 2000. ETSAP contracting parties get a small group license for free, and others can purchase it from KanORS. Access to technical support and updates is subject to an annual maintenance fee (20% of the initial cost), after the first year.

Overview of the VEDA system for TIMES modeling

VEDA2.0 is a powerful tool required by complex mathematical and economic models for handling data input and smart exploration of the results created by such models and the creation of reports.

Data and assumptions are fed into VEDA that provides input to the TIMES code. VEDA accepts input from a variety of Excel files with different (flexible) structures that are tailored to work efficiently with data intensive models. The TIMES code works in the GAMS environment and produces text output that is read by VEDA. VEDA produces numerical and graphical (mainly via Excel) output for the user.

The system is fundamentally biased towards large-scale multi-region models. It can certainly support small single-region models, but its real power comes into play when modelling several regions together. The primary focus of this system has been on expanding the envelope of possibilities. VEDA uses a support site and a forum to support its users.

Architecture

All input data resides in Excel workbooks. XLSX/M format is recommended for Veda2.0. Modularity is one of the core features of Veda. This is to make major reconfigurations possible and efficient. This also makes it easier for multiple people to work on different parts of the model in parallel. This is achieved by segregating the input data into the following sections:

• Core definitions of regions, timeslices, modeling years, and commodities
• Technologies with existing stock
• New technologies
• Demands
• Trades
• Additional parameter definitions for technologies and commodities

There can be multiple files for each type of data, apart from the first one – the core definitions. In each model folder, these files are organized in the structure shown below.

Files expected in these sub-folders are as under:

• Root folder has SysSettings (core definitions), Base-year templates (existing techs), and set definitions.
• SubRES has files with new technologies
• SuppXLS has the scenario files (additional parameters (or modifications) for all existing and new technologies and commodities)
  • Demands has the DEM_Alloc+Series to allocate drivers to demands, and ScenDem_ for driver scenarios.
  • Trades has ScenTrade__Trade_Links for defining trade links, and ScenTrade_ for declaring attributes for trade processes (which can also be done in regular scenario files).

Veda2.0 is a C#.NET application that reads these Excel files into a PostgreSQL database, offers tabular and graphical views of the data as TIMES parameters, and submits the data to the TIMES code.

Versions

There are three different versions of Veda2.0:

• Academic
• Standard
• Advanced

The academic version works on a single core, but is still much faster than VEDA_FE/BE. Standard version uses multiple cores for certain operations, like processing FI_T and DINS tags, and writing DD files. In smaller models (academic use), the difference would be imperceptible. Advanced version has two additional features – Collaboration, and Reports.

Collaborative working

Multiple users working on the same model on a server will be able to share the following:

• Model runs
  • Runs from multiple users, even with the same name, will be usable in the Results module. “User” will be a dimension in the data, like region, scenario etc.
• Input Data GDX
• Results views definitions
• Various groups and case definitions for Run Manager

Further, the JSON files in Appdata folder will also retain username information. So, users sharing model folders will be able to use or filter out groups, cases and views created by other users.

Advanced reporting

VEDA_BE and the Results functionality in Veda2.0 work well for interactive and production reporting. But there are two limitations, removing which can make this a lot more powerful and flexible. First, the reporting variables are trapped in tables – we don’t have direct control over them. Second, we cannot add dimensions to the output views – we are limited to process and commodity sets in terms of segmenting the output beyond the native indexes like attribute, region and time. Let’s take transportation final energy (in a rich model like the JRC_EU-TIMES) as an example: I want to see energy consumption by scenario, region, fuel, mode, size, and technology. Scenario and region are separate indexes, and fuel can be managed with commodity sets. But we have only process sets to deal with mode, size and technology. The entirely new approach of custom reports uses an Excel template to define reporting variables in a very efficient manner, and freely add dimensions based process/commodity names, regions and scenarios. Further, it is possible to include exogenous data in this process. It can be used to include historical energy balances to show historical trends in summary views, and to set up calibration checking views.

Licenses are priced as per institutions as well, like before. Academic version is accessible only to degree-granting institutions.

Key enhancements over the legacy version

Broadly, there are two differences between the old and new versions of Veda:

1. VEDA_FE/BE were based on VB6/MSACCESS and Veda2.0 is on C#.NET/PostgreSQL.
2. Most of the features in old versions of FE/BE were built incrementally, over 20 years. All these features (and many more), are a part of the fundamental design in Veda2.0.

Ease of migration and use has been kept in mind

• Much smoother installation process.
• Very stable application and reliable data processing – No Sync surprises.
• Results handling integrated in the same application – Veda2.0 replaces VEDA_FE + VEDA_BE.
  • Much faster views processing.
  • Sets and view definitions can be migrated from old SnT MDB files.
• Works with practically the same model input files (Excel templates) as VEDA_FE. No change in file naming conventions or tags.
  • The few changes that are required are clearly documented. Clean models can be migrated within hours.
  • Automated migration of set definitions and results views from Veda_SnT.MDB files makes migration a lot easier.
• No limit on length of item names (process, commodity, UC, commodity groups).

Veda2.0 – enhanced features

• Various browse features, run manager, and even navigator, work very independently and can be used concurrently. Multiple models can be used concurrently.
• A new Start page makes it easy to work with models, also with different branches on Git. There is a section that pulls information from the Internet – to be used to display tips for users.
• All pivot grids have CSV export facility, which is very useful for creating input for visualization tools like Power BI and Tableau.
• Unit conversion is more advanced.
• Possible to write GAMS instructions in different locations of the RUN file and top or bottom of DD files.
• Milestone years can be specified directly, instead of using period lengths.
• All forms are extremely independent and allow very flexible layout changes. Users can continue using other modules even when the DD files are being written or the model is synchronizing.
• Three to ten times faster synchronization, depending on the model structure and number of cores available.
  • Guidance provided for reducing the processing time further.
• DD writing is an order of magnitude faster and scales directly with number of cores.
• Smart filtering available throughout the application.
• All data is rendered in a pivot grid for browsing, like before, but the pivot tool is much improved.
  • Handy charting facility available with all data views.
• Interdependence across scenario files (due to FILL/UPD/MIG tables) is tracked and reported.
• Column position of any tag, including FI_T and UC_T, is not important anymore, making it less error prone.
• Comprehensive documentation of all tags and columns supported by each.
• Set definitions are shared by input and results sections and it used to be difficult to keep things in sync. Now the sets file is synchronized seamlessly by both functions.
• Powerful sets playground feature allows interactive viewing, editing and creating new sets, which are automatically updated in the set definitions file.
• Open architecture: all user definitions like scenario groups, cases, results views etc are stored in (to be documented) json and CSV files. In principle, users can modify these files programmatically.

Highlights of Veda2.0 – under the hood

• Veda2.0 uses C#.NET for UI and PostgreSQL as the back end.
• Based on the MVP (Model View Presenter) architecture, which makes it very stable, and makes maintenance and further development relatively easy.
• State-of-the-art version control processes in place for the source code.
• Each release undergoes elaborate testing.

Veda online (VO)

Veda online (VO) enables working with TIMES models via Internet Browsers. Veda model folders are expected to be on GitHub, and all the core functionality of Veda2.0 – Synchronizing Excel files, browsing input data, running models, and looking at results/reports – is available online.

VO users can make their models accessible to other VO users with control over what modules are visible and the downloadability of input/output. This makes collaborative development much more efficient. Further, this opens the possibility to disseminate models – to create model users.

It offers several advantages over Veda2.0 to individual users as well:

• No software setup/updates needed
• Enforces version control discipline via GitHub
• Model synchronization and runs on state-of-the-art servers
• Availability of model input/output no longer dependent on specific machines
• Far superior data visualization in Reports, Results, and Browse.

While I can manage incremental modifications with VO, Veda2.0 remains my preferred tool for model development and debugging.

VO is an open website. Public models are visible to visitors even without logging in. An annual subscription is required to create models and perform runs. Making model runs also needs GAMS/CPLEX cloud service, but that is managed by KanORS.