TRAME (TRAnsmission of MEssages)[1] was the name of the second computer network in the world similar to the internet to be used in an electric utility. Like the internet, the base technology was packet switching; it was developed by the electric utility ENHER in Barcelona. It was deployed by the same utility, first in Catalonia and Aragón, Spain, and later in other places. Its development started in 1974 and the first routers, called nodes at that time, were deployed by 1978.[2] [3] [4] [5] The network was in operation until 2016 (38 years) with successive technological software and hardware updates.
In 1974, packet switching was a technology known only in research circles. The concept began in 1968 in association with the United States' Advanced Research Projects Agency (ARPA) research project ARPANET. The idea of applying the packet switching concept to electric utilities control communication networks first appeared in 1974 when the Swedish power utility Vattenfall started to create its TIDAS packet-switching network[6] [7] [8] and was followed by the Spanish electric utility ENHER,[9] which aimed to telecontrol and automate its high-voltage power grid. For this purpose, ENHER created a specific team of people to develop both the packet-switching network and the supervisory control and data acquisition (SCADA) system, also called the telecontrol system. By 1978 the first four TRAME routers were available and by 1980, eight of them were deployed and operating.[10] The printed circuit boards (PCBs) controlling the communication lines were connected to a shared memory PCB allowing them to exchange data and messages. The project was developed together with its main initial application, the Telecontrol or SCADA system SICL [11] [12] [13] with which initially they shared a very similar hardware. The maximum link capacity was 9600 bit/s, which in 1980 was the maximum possible on a 4 kHz wide voice channel at the time. These channels were the basic unit of the then-analog communication systems in use. By that time power utilities used either telephone calls or low speed (below 1200bit/s) dedicated links for telecontrol, typically shared among ten high-voltage electrical substations.
The basic service provided by the TRAME network was SCADA or Telecontrol to automate the high-voltage power grid, thus improving operational efficiency, which was until then operated manually with telephone communication between human operators. Each TRAME router was associated with one or more remote terminal units (RTUs) of the SICL telecontrol system. It also had connected screens, and later PCs, located in electrical substations to interchange messages between them and with the Control Center[13] located in the well-known in Barcelona. It was a kind of predecessor to today's e-mail. Later, in the 1990s, other protocols (X.25, IP) were developed to include corporate information technology (IT) terminals, company physical surveillance systems and other services. Additionally, applications and terminals were developed for the transmission of voice and video over the TRAME network.[14] [15]
The TRAME routing system,[3] like that of the original ARPANET, was based on the Bellman-Ford algorithm but with "split-horizon"[7] as in the Swedish TIDAS network, but with an original improvement.[3] This protocol allows optimal paths to be found in meshed networks for each packet to be transmitted, allowing the shared use of the same network by multiple services. In contrast, traditional circuit-switched technology used to establish dedicated circuits for each service or communication. The addressing of routers and terminals used a proprietary system with a 16-bit address; it would be the equivalent of the well-known IP (Internet Protocol) version 4 (IPv4), still in use on the internet today, which uses 32-bit addresses. It is necessary to take into account that in 1978, the IPv4 protocol did not yet exist since the IPv4 version used on the internet did not appear until 1981, and in fact, did not reach the general public until much later.
The line protocols were also proprietary and were called UCL ('line control unit'), which linked the routers together, and UTR, the access protocol. They were designed to offer the highest quality of service required by the telecontrol/SCADA function in terms of data integrity and availability set by the International Electrotechnical Commission (IEC) IEC-870-5-1 and ANSI C37.1. standards, and because the protocol used at the time in corporate computer networks, HDLC (high-level data link control), did not offer enough quality for critical industrial applications.[16] [17] Later on, other protocols like X.25 and IP were also made compatible with the aforementioned TRAME protocols. In 2000, the UTR protocol was replaced[18] by the international standard IEC 60870- 5-101/104.
Initially network flow control was based on the management of eight data priorities in head-of-the-line (HOL) waiting queues. Later and after some experimentation,[19] a flow control method based on a bit indicating route congestion and management of the gap between packets when accessing the network was adopted. This required measuring the capacity of the route bottleneck.[20] [21] [22] [23] An end-to-end protocol was also added for some flows requiring order preservation like X.25.
To last for 38 years, the technology had to endure intense evolution. There were essentially four TRAME generations which are summarized in the table.
| Concept | TRAME 1 | TRAME 2 | TRAME 3 | TRAME + | |
|---|---|---|---|---|---|
| Development started | 1974 | 1993 | 1998 | ||
| Deployment started | 1978 (4 routers in operation) | 19881990 (fully deployed) | 1995 | 1999 | |
| Processor | i960CAi960RM | ||||
| Maximum link speed | 9600bit/s | 64kbit/s | 2Mbit/s | 10Mbit/s | |
| Hardware | 16 processors sharing a common memory | 16 processors sharing a common memory | 16 processors sharing a common memory distributed among the 16 PCBs. | A single processor with a shared multimaster 1Gbit/s bus to communicate I/O boards. | |
| Hardware developer/manufacturer | ENHER/ISEL (INI) | ENHER/ISEL (INI) | DIMAT, S.A. | DIMAT, S.A. | |
| Addressing field | 14 bits · 256 routers (8 bits) · 64 terminals per router (6bits) | 16 bits · 256 routers (8 bits) · 64 terminals per router (6bits) · 4 interconnected networks (2 bits) | 16 bits, as in TRAME 2 | 16 bits, as in TRAME 2 | |
| Managed priorities | 8 | 8 | 8 | 8 | |
| Inner link protocols | UCL (proprietary) | UCL improved for 64kbit/s (proprietary) | UCL improved for 64kbit/s (proprietary) | UCL improved for 2Mbit/s (proprietary) | |
| Access link protocols | UTR (proprietary) | · IPv4 (1999) · X.25· Proprietary: UTR | The same as for TRAME 3 | ||
| Network size evolution | 1978: 4 routers1980: 8 routers 1983: 27 routers 1987: 32 routers | 1994: 50 routers | 2004: 222 routers2014: some 3000 routers (several networks) | ||
| Router appearance |
A description of the four generations of TRAME is provided below.
The project began in 1974 and in 1978 a first network with four routers was already installed and in operation at the electric utility ENHER. In 1980, the network had eight nodes in operation (see Figure I). The hardware was based on the Zilog Z80 processor and had a multiprocessor structure with 16 processors sharing a common memory. The software was developed at ENHER's headquarters located in the well-known Casa Fuster,, 132, Barcelona, using the Z80 assembly language. Beyond 1980 the software began to be written in C programming language and an HP64000 Logic Development System emulator was used for the purpose. The hardware was produced by ISEL, an INI company.
The routing system was a variant of Bellman-Ford with split-horizon. It was an improvement of the original ARPA network routing system consisting of an original update procedure which allowed for a faster reaction to changes. The distance function was the number of packets in the output waiting queues plus one.
The line protocols (UCL for internal lines linking routers and UTR for accessing the network) were designed to meet the stringent requirements set for telecontrol (SCADA) of high-voltage power networks (IEC-870-5-1 and ANSI C37.1 standards).
At the OSI transport layer, windows with a width of 1 to 8, depending on the required service, residing in the terminals were used.
Initially, addresses were only 14 bits long to address both the routers (called nodes by then) and the devices connected to them. They were made up of two fields, an 8-bit field to address the router and a 6-bit sub-address to address the terminals connected to it. The node address was assigned to the nodes and not to the ends of the links as in the internet.
The basic advantages of TRAME over other technologies used in electric utilities at the time were in part due to the packet technology itself: ability to manage any network topology, automatic adaptability to topological and traffic changes, integration of different link technologies (digital or analog) and capacities in a single network, open and decentralized intercommunicability between users and devices, simultaneous communication with several users and locations from a single physical connection, and integrated network supervision. In fact, the network was provided from its inception with a supervision center[24] consisting of a computer and a synoptic board located at the company's headquarters (see Figure II).
But other advantages were due to the specific design of TRAME: high data integrity, priority support for packets, and ease of including special protocols such as the many SCADA protocols in use at that time. All of the above resulted in improved quality of service, especially with respect to data availability and data integrity, and in the integration of services in a single network. Part of the evolution of its deployment can be seen in Figures II to IV.
In 1990, TRAME 2 was fully deployed and TRAME 1 was replaced. The processor of the new hardware was Intel 80286 and the hardware structure and external appearance of the routers was very similar to that of TRAME 1. The software was written in C and the above-mentioned emulator continued to be used.
Improvements over TRAME 1 were the introduction of the standardized X.25 access protocol to enable the connection of corporate terminals to the network, the ability to handle the 64 kbit/s of the new digital lines, increased switching capacity, and the introduction of an end-to-end protocol to avoid packet loss and clutter as required by X.25.
An important improvement was the possibility of using dual homing to increase terminal availability; they could be connected to the network by two access points. For the purpose, the terminals had two addresses, a primary and a secondary one.
Regarding addressing, in 1991 two bits were added to the addressing to indicate the network. The address space was thus increased to 16 bits and, in this way, up to four networks could be freely meshed as in a single one. This addressing scheme was maintained in subsequent versions of TRAME.
The hardware was again a multiprocessor structure with 16 processors sharing a common memory but the latter was not a separate PCB but instead was distributed among the 16 PCBs to avoid single points of failure. The interconnection of PCBs was done with a shared 40 Mbit/s capacity multimaster bus designed and manufactured by DIMAT, S.A.. It also included a serial channel for maintenance, monitoring, reprogramming and resetting of the different modules through a terminal connected to them. The software was developed by ENHER in collaboration with DIMAT, S.A..
The routing algorithm remained the same, but the distance function was changed to a less dynamic one. A flow control procedure based route congestion metering and backwards indication to the source was introduced.
Improvements over TRAME2 were IPv4 support, the introduction of an SNMP monitoring agent, a new flow control system, an improved distance metric that made the system less dynamic, and an autoexec task to periodically check hardware and software.
The hardware design was radically modified by moving to a single processor per node architecture as opposed to the traditional TRAME hardware. It had two alternative base modules of different capacity based on Intel i960CA and i960RM processors with a 1 Gbit/s bus to communicate the different router boards. The number of physical interfaces was only ten (eight serial + two Ethernet (10BASE2 or 10BASE-T)) since Ethernet allowed for the connection of several devices on a single LAN. It also had a front service serial channel. By losing redundancy (a single processor per router) the node lost some availability over previous versions of TRAME. This was done for economical reasons stemming from the fact that the network was being extended to smaller substations where cost constraints are higher. Dual homing could help in places with more stringent availability requirements.
Improvements over TRAME 3 were the ability to handle 2 Mbit/s capacity links, smaller and less expensive routers, access by Ethernet and standard protocols, and the change from the proprietary UTR protocol to the internationally standardized ones for SCADA systems (IEC 60870-5-101 and IEC 60870–5–104) with an original adaptation to packet-switched networks.[25]