Numbers can say a lot about things, but they can equally be very abstract. Having something to look at can be very helpful in understanding relationships between data points and quantifying the significance. In most cases a quick trip to your favorite spreadsheet editor can help with that, but there are also ways to extract and visualize data in other ways. The methods are about as plentiful and colorful as you can imagine the data will be, so without going into specifics on which tool to use we can take a more general approach to what might be of interest. Selecting the right tool for data extraction and visualization depend on a lot of factors, so it makes little sense to provide any syntax as example, when, due to the nature of the data itself, the approaches different tools might take can be vastly different.
This is even more true for the data contained in things outside of databases and other storage systems. Extracting data from more dynamic sources is a completely different task in data processing, which can require different tools and a much broader approach to interpreting the results. Diving into realms that deal directly with data provided by other software and the resulting uncertainty of both accuracy and validity can mean downright manual review.
Visual data
A bit of a different matter is the data used to create the actual visuals you can see. Most of that can be easily viewed easily and without the need to convert anything. This is less the case for the information that generates parcels. The data required to define the outline of them is setup as a bitmap, which is a short way of saying a large grid of numbers, which simply defines each part that belongs to a particular parcel. Sounds easy enough to understand and it is from the perspective of a computer, but in order to convert that into something visual for a human brain requires a bit more. It is complicated by the fact, that the data contained in the bitmap is stored in a format not easily readable as actual bits of 1 or 0. Instead the data, as any data is on modern computers, is put together in bytes, which first have to be broken up into their 8 parts to return the bits we need.
Once the data is available as simply a grid of numbers the fun part begins. Each entry in the table constitutes a single parcel, so multiple entries each form the lines between those parcels. In order to visualize this by way of showing a parcel as a differently colored area on the square making up a region, the various entries have to be combined back into a data structure that represents an assigned of a parcel identity to each occupied spot. Effectively replacing the grid of 1’s and 0’s with the id of the parcel. With that part complete the hard work is out of the way and what is left is simply presenting the resulting data structure visually, which can be accomplished in more ways than there are grains of sand.
As an example it can then be shown in this form:
Additional information can be gathered through other tables within the region context to show things like the teleportation target or specific data about the parcel itself. Even the rough locations of objects and their size can be visualized. There really only exists one limiting factor and that is data processing requirements and thus speed. Transforming data from its numerical representation inside the database to visual objects can be quite resource intensive depending on where the processing occurs. There are tradeoffs in sending data for processing by whatever is viewing it, to processing it at the server end and providing only the object data necessary for the visuals.
In a similar fashion the terrain tables contains information about the heightmap used to generate the visual terrain on a region. It, too, is stored in a format that isn’t easily readable. In difference to the parcel information, the data structure here contains not simply 1’s and 0’s, but floating point numbers. It makes sense in that the terrain heightmap requires a lot more granular information than can be represented with even whole numbers. While a grid of complex numbers at first does not sound much more complicated to decode than whole numbers, what compounds the difficulty is that these numbers are stored in a specific bit format in order to provide more digits after the comma. Commonly with image files, which is what ultimately heightmaps are, this is denoted by the bit depth and can range from 16 all the way to 32 bits in most cases. Consequently a completely different approach is required to turn those bits back into humanly readable numbers or a visual representation that is easily viewed and understood.
Fortunately in regards to visualizing this, the routines required for converting such a bitmap to an actual image are a dime a dozen and most programming languages provide tools for this. It becomes more a manner of working out which one provides the faster conversion as image processing speed heavily depends on what hardware ultimately does the conversion. This is because when it comes to floating point numbers there is one part found commonly in most computers that is leaps and bounds better and wrangling those than anything else. A graphics card, which as the name suggests is rather well suited to dealing with images and graphics. So an easy choice, so long as it is actually available, which is not the case when it comes to the hardware configurations found for applications that are meant to provide something to hundreds of people. Beyond the availability itself, the interfacing with the hardware can be quite tricky and even though software exists to ease the burden, it often requires to interface with something that isn’t easily adjustable should the hardware change. This leaves the processing to the main processor, which fortunately is still no slowpoke when it comes to required transformation of data.
OpenSimulator ultimately provides access to this data in a visual form through the utilities in the terrain section. They can be used to turn the data into images, which given the right software, can be viewed and manipulated. This also means, the required code is already readily available through the source code itself. Thus anyone that wishes to create a visualization has a direct point of reference on how to accomplish this. Granted it still requires some changes or a translation into a different programming language, but it is certainly easier to read compared to programming code used to visualize the parcel information in whichever viewer is used.
On the higher grid level the visual data is much less impressive, but nonetheless interesting. Beyond the usual suspects of user profiles and the data making up the search functionality, we can also visualize the world map. The code responsible for showing the world map in a viewer requires a data structure that appears to be quite conducive to visualization, but the devil is in the detail. Much like most maps, it would not make sense to show it as one large picture, because this would take up a lot of resources. Instead the map is split up into tiles, that can be individually retrieved only when necessary. This is normally a solved problem still, but unlike the maps of our own planet the way each tile is arranged follows a different path. This means that any software normally designed to handle earthen maps will arrange the tiles in a manner that changes their orientation. This is because the coordinate system used extends from a different origin point and moves across the screen in a different direction.
Changing the way a coordinate system is interpreted and either building the map based on it or transforming the coordinates back to the standard is easier said than done. As this is likely not the first time this type of problem has come up, depending on the source of the map tiles, most software designed to provide general maps of anything will have some way to make the changes needed for this. Though this still means significantly more work is involved than just simply pointing the software at the tiles and enjoying a lovely map in front of your eyes. With this part completed, however, there is not much else left beyond adjusting the way in which different scales are handled to provide the all important zoom functionality. Similarly to parcels we can reference other information in the database to add more information, such as the names of visible regions and even who owns them or how many avatars are currently using them. A complete analog to the viewer world map is possible through utilizing the data providers of each region and the information contained in various other tables. Although this sort of data processing is likely to require a lot of consideration in regards to frequency of refresh or caching mechanisms, especially if there are a lot of regions to process.
A good example of the possibilities with relatively simple processing can be seen on our in-house grid ZetaWorlds through the world map provided there: https://zetaworlds.com/map
Hidden data
Might not be the best description for the information contained in log files. OpenSimulator does not store historical information in the database, otherwise it would easily exceed the storage capacity of most installations. Information about what was can only be retrieved through the log files or strategic logging of information in the tables to other tables. As mentioned above in the technical section this can be quite useful for analyzing trends or detect the causes of issues that arose over time.
Specifically in regards to log files there are two main points of information. The region log files mostly containing information about critical actions performed on a given region as well as those that are responsible for them. On the other side the logs generated by the grid side provide a broader overview, but nonetheless critical information when it comes to issues.
Parsing this data into a usable format can be difficult depending on what tools are available for this. A task made more complex due to variation in the data as there are parts directly controlled by a user, so possess the potential to be almost anything. Searching for keywords and patterns or directly turning the whole thing into data stored in an actual database are valid options and depend on what sort of goal the data retention is supposed to provide. In most cases the mentioned searching through the logs is likely to be enough, but storing critical pieces of information in a database can be both simpler and faster to search through as well. So in the end this strongly depends on both the use case and the simple volume of data generated.
OpenSimulator, for the most part at least, does utilize standard logging techniques that should provide ways to interface with the generated logs. Through the use of those standard libraries the logging can also be changed to provide additional information or reduce the logged data for easier and faster searching. Most of the available changes to the logging setup are documented within the documentation available for the log4net library most of the logging depends on. Beyond this there are specific additions to this added to handle certain requirements OpenSimulator had in terms of logging that are not covered directly by log4net. Unfortunately documentation for those additions is lacking, which means most of the information regarding its operation and capabilities can only be obtained through reading through the relevant source code.
This about covers the most interesting parts of the data within OpenSimulator and how to extract some of it. There are a lot more things that can be interesting to look at depending on your use case and not all interesting data might be logged or readily available either. Fortunately either through existing extensions or by changing the code directly almost everything can be retrieved, logged and processed for visualization.