Stigmergy in Computing Systems

Data in file systems, OS registries, databases, wikis, and social media sites are to computing what pheromones are to termites and ants.

Biological and digital information processing have much in common, including the way multiple independent entities collaborate and cooperate by sharing information. Direct real-time communications (nerve signals and point-to-point IP packets for example) are appropriate for rapid and precise cooperation. Broadcast messages (e.g., hormones such as adrenaline and broadcast IP packets) are appropriate where multiple unspecified entities must respond promptly and perhaps differently as appropriate to their specialized function. But many sorts of collaboration are not anchored to specific temporal circumstances. They may instead be associated with specific physical locations, e.g., a mobile device is inside/outside a firewall or, a traffic light controller is located at a specific traffic intersection. A collaboration may relate to a specific activity such as inventory control for a particular chain of stores. Those sorts of collaboration can only be facilitated by information deposited in appropriate data stores accessible to the computers involved.

Digital stigmergy relies upon data repositories populated by, modified by and queried by a distributed community of agents. In single computers, file systems are the the most familiar examples of stigmergy structures. Others include:

In multicellular computing, stigmergy structures provide a persistent and reliable means of asynchronous communication that supports group collaboration. Familiar examples in corporate intranets and in the Internet include databases, file servers, DNS servers, email servers, Web servers, search engines, social media hubs, and all manner of other stigmergy structures. Consider for example a customer database for a corporation. Its model is typically a relational data model of customers, their orders, and their accounts. It is modified and read by sales personnel, the shipping dept., accounts receivable personnel, and perhaps to some extent by the customers themselves if the company provides a Web interface for orders.

Some persistent digital structures intended for one purpose have evolved organizing roles for quite different collaborations. The most notable example is Twitter, which originated with the idea of providing a persistent common data structure populated with short cellphone SMS text messages that could be accessed asynchronously by its users at their convenience. But it evolved into far more than that. Hashtags, for example, emerged spontaneously to organize collaborative discussions. But Twitter hashtags may also be used to coordinate a Distributed Denial of Service (DDoS) attack or a flash-mob revolution in Egypt! And the hashmark (#) was an arbitrary signal of relevance. Other equivalent symbols or strings can be created for ad hoc group communication and the "posters" and "readers" may be bots signalling something relevant only to their programmed purpose.

The most economically significant examples of networked stigmergy structures are the financial exchanges that support high-speed computerized trading in stocks, bonds, options, commodities, futures, and more arcane financial instruments. The New York Stock Exchange, the Nasdaq, and Europe's BATS are the best known exchanges but there are other much faster specialized exchanges. They depend upon large high-speed databases into which trading bots insert trade orders. The trades are consummated without human help. High-speed trading systems routinely execute trades in less than a millisecond a few claim less than 40 microseconds and one boasts of 987 nanoseconds. They are driven by computerized decision algorithms based upon the most recent trades. The data repositories supporting the trades are the stigmergy structures owned by major markets such as the NYSE. These markets charge the high-frequency traders to allow the traders to place their trade-bots as close as possible to the market servers; every foot the bots are away from the market server adds a nanosecond delay. The profits available to the fastest trade-bot can be large. The economic consequences of high-frequency trading on the wider economy can be far larger. The May 6, 2010 "Flash Crash" market meltdown, where the markets lost $1 trillion within fifteen minutes, apparently resulted from unforeseen interactions within and between these automated exchanges.

Novel stigmergy structures have emerged in the World Wide Web. These include widely distributed data in peer-to-peer (P2P) file sharing networks such as Bit Torrent, in Facebook and EBay, and in "Edge-of-the-Web" page caches such as those provided by Akamai and Google's internal replicated caches. The largest Internet stigmergy structure is Google's distributed crawler index paired with its page-rank data and AdSense ad-server system.

Computing stigmergy structures are fundamentally different from biological stigmergy structures in that they occupy a world of bits rather than atoms. Since physical stigmergy structures must obey the laws of 3-dimensional space, they provide coordination of location, shape and proximity and they prevent interpenetration. The location and shape of physical/biological stigmergy structures determine to a great degree what sorts of interactions can take place between the participating organisms. In contrast, stigmergy structures in computing systems are embedded in various network topologies determined by rules that provide logical structure to the various data repositories. So in most cases we can only imagine rather than see the "shape" of digital stigmergy structures. One exception is the stigmergy provided by simulated 3-D environments, such as Second Life or World of Warcraft. that create simulated visible worlds maintained by the game's "physics engines" that expend substantial compute resources to maintain positions and shapes, and to prevent interpenetration.

Multicellular stigmergy structures and corporate IT

Most complex computing systems are designed to support persistent human organizations such as corporations, universities, or governmental agencies. Corporations are not only a type of collective “self,” recognized in law, but also are a “self” organized by stigmergy. The people in the company create external structures including buildings (offices, factories, warehouses, etc.), equipment (ships, trucks, milling machines, desks, copiers, and computers), records (the “books,”, the contracts and other documents), and persistent financial structures such as bank accounts and shares of stock and bonds issued by or owned by the corporation. Increasingly today the most important structure in a corporation is not the bricks and mortar, but the IT infrastructure – the physical and logical networks (VPNs), databases, and applications that manage and transform business-critical information.

Because an organization's digital stigmergy structures are vital to its function, their security is vital to the survival and competitive strength of the organization. Therefore, the security of these structures has become the major focus of IT organizations as they attempt to protect the computing portion of the corporate "self."

“Selfness” in computing systems is often misconstrued to be about the identity of the connected leaf devices and the identity of authorized users. Yet the identity of a given machine can’t be the determining factor since machines can be lost, stolen or compromised by a virus or worm. Nor can the identity of the person be the determining factor since people move from company to company, they forget their passwords, they leave their passwords around on sticky-notes, or they choose trivial easily guessed passwords.

For these and other reasons, we are beginning to recognize that the perimeter of an institutional network, i.e., the collection of PCs, iPads/Pods/Phones and other personal devices used by employees, is ultimately indefensible. Moreover, the perimeter is also indefinable because it intersects with supplier and customer systems. And the corporate employees themselves work remotely, sometimes in a disconnected mode. Yet, just inside the fragmenting perimeter lies the core of the corporate infrastructure, i.e., the definable and defensible IT infrastructure comprised of the network, databases, and institutional application servers. That core stigmergy structure is the corporate IT “self.” The fundamental job of the staff in an IT organization is to maintain this vital corporate computing infrastructure. They play a role similar to ants maintaining the nest, or bees maintaining the hive.