Farewell 2016

As another year comes to close Zetamex Network is moving into 2017 with some fresh plans and changes that we hope are as exciting for you as they are for us. Among the changes are the official launch of Zetamex.solutions, our custom OpenSim and Web-Solutions branch aimed at those who need something of their own and no run-of-the-mill packages. Also on the horizon are some new projects we are working on at the moment and that will see the light of day in the coming year.

To go into some closer detail, as previously announced we are changing billing to better reflect the scope of our customers projects and adjust pricing according to their needs and budgets. This now also brings a change to the currency we use. So far we have billed in USD, a currency widely used on the internet and international markets. However USD is not the currency we file our tax forms in and as a result each invoice has to be converted to EURO before we can file taxes. With the recent fluctuations on the stock markets the value has jumped up and down resulting in an immense workload of our financial department, converting each little amount according to historical data just to file a “handful” of taxes. This additional workload is time consuming and unfortunately time is still money. With USD and EURO only being a few percentages apart from one another we have elected to eliminate this workload by switching the billed currency over to EURO. This will significantly reduce the time spent on our end of billing and thus reduce the prices in the long run. Our payment gateways will automatically convert the invoiced amount from EURO to USD for you at no additional fees.

With Zetamex.solutions bringing fully custom OpenSim solutions to the market we did not want to neglect the entry-level side of OpenSim. People who just want to try out OpenSim or lack the budget for a large custom solution will be happy to hear that we have been working on something that will make OpenSim as easy as ordering pizza. OpenSim.hosting will be our new, standardized, fire-and-forget, one-stop-shop for your drive-thru OpenSim needs. All cheesiness aside, OpenSim.hosting will feature a simple to use interface allowing you to easily spin up a simulator and manage almost all aspects of it. You will be able to choose from a host of new features we have been rolling out to our custom solutions in the past along with daily billing, for when you just want to host an event and need some extra space. We will be going into more detail about the features of this new system in the coming months as we continue to develop and refine it. We currently do not have a specific launch window and will most likely soft-launch the system for early adopters. Stay tuned for more information as 2017 progresses.

Nope, not done yet, we got more to share. Another big construction zone over the last couple months has been ZetaWorlds, our in-house grid we have been using both as beta-grounds for new features as well as space for refugees from other grids. ZetaWorlds will continue to be a major focus for us next with new features being added and sales of land opening up as well. Our aim for ZetaWorlds has always been to provide a stable platform for those looking for a new home and to hypergrid and explore the metaverse. For a couple months now we have been developing features for both the grid and website to finally start engaging the community of ZetaWorlds. For that matter we have signed up a Community Manager that will help us build the community and work together with you, the resident, to bring better service to you and create the events you would like to see. Give a warm welcome to Edison Rex(Paul Harriman) your new address for anything community related. Edison will also take over the ZetaWorlds specific blog and post updates and events there so stay tuned for that.

That was quite a lot of information all at once, we will give you some time to digest it all over the holidays and already wish you a happy New Year and thank you for your continuous support throughout this year!

Introducing Zetamex.Solutions

Zetamex Network has offered a host of standard packages for OpenSim in the past. That is all well and good, but sometimes standard run-of-the-mill solutions are not what you may be looking for. For that purpose we have created Zetamex.Solutions specifically aimed at those who want to make more out of OpenSim. Our existing customers already enjoy many of the perks this branch of our company offers. Now those are available for everyone.

Along with this we are changing part of our billing system to better serve to the different budgets and projects people approach us with. This means removal of the standardized packages to be replaced with flexible, budget-based billing. Further are we streamlining our support system removing the need for the Dedicated Support package. Instead support is given equally to everyone and special requests are met with more transparent billing system. These changes were made after the feedback we received over the last couple of months has shown that they were needed. We hope you are as excited about them as we are. We will continue to gauge the feedback we receive to improve the quality of our products and the support we give for them.

PHP-SAS Performance Analysis

Since our release of phpSAS into production we wanted to explain a bit more how it works. Specifically in terms of the performance differences.

To start off, phpSAS is much like SRAS in that it is stores files to disk directly, then organizes the data using SQL. It was built to support both PostgreSQL and MySQL/MariaDB so that an existing SRAS database can be dropped in without any issues. Furthermore, we have noticed that PHP 7’s ability to query is far faster than that of Ruby 1.9.3, which was the last working version that could run SRAS. So in simple terms, a request comes into phpSAS which then queries the SQL database which in turn returns the information and location of the asset, then serves the asset back to the grid/simulator/user.

Since running it in production on several different types of grids, low to high load environments, we wanted to report exactly what we are seeing on our end in terms of the performance. First off let’s look at the raw statistics.

Grid Active Users Number of Assets Memory Usage (phpSAS) Memory Usage (SRAS)
37 902,113 (84.2 gb) 32mb 215mb
135 534,052 (48.7 gb) 48mb 1.2gb
284 3,308,648 (148.9 gb) 54mb 3gb

As you can see from the data above, the statistic speaks for itself. The new asset server is able to scale significantly higher, and with the ability to put it behind a load balancer allows it to handle even higher spikes of traffic. The reason we have not included CPU load is because there is really no significant CPU load to speak of. Both SRAS and phpSAS never really spiked above 2 percent load even during heavy traffic of loading IAR’s or OAR’s from multiple simulators.

In terms of end-point performance for the client the following chart should give you some idea of the performance compared to the other asset servers. Please note that mileage may very on these results depending on hardware and the grid’s connection. We tested this with fairly simple mesh, textures, and objects. The Viewer was run on a machine with 8gb of ram and a Nvidia grid card on ultra-settings, while on a 1Gbps connection. Provided by LiquidSky.

Stock FSAssets SRAS phpSAS
Mesh 4.12 1.55 1.89 1.31
Textures 2.07 2.01 2.15 2.01
Objects 2.81 3.01 2.67 1.41

If you are interested in seeing the difference for yourself then head over to ZetaWorlds and create a local account. While you could use the hypergrid, you will most likely not see a great difference in performance, as load times greatly depend on the performance of the grid you are teleporting in from.

Replacing SRAS

Zetamex Network has been a big user of SRAS for a long time. We didn’t make the switch to FSAssets due to the performance to system resource usage was still much higher than that of SRAS. Since the most recent releases of many mainstream Linux distributions it is becoming more and more noticeable that SRAS is showing its age. We had already been using software to allow for SRAS to run on older versions of its dependencies in order to get it to work at all.

We realized this was just not cutting it anymore, and we really needed something new to replace the outdated SRAS and make sure that we could support it for the future to come. Reaching out to freelancers to assist us in the project in converting or rewriting something backwards compatible to SRAS, as FSAssets is not directly backwards compatible but could be migrated with some effort. After shopping around for freelancers we came upon one already deep inside the OpenSimulator community who has done custom work for other grids, with whom we have worked with in the past on a few smaller projects for some of our clients.

We decided on to use PHP for the replacement, as we have already built almost the entire Zetamex Network back-end on it. Furthermore, we made sure it was built without relying on any dependencies other than PHP itself. This made it slim, lightweight and easy to future proof by having it all written from scratch for easy upgrades in the future. The development took a while, but this past week it has been rolled out and is running on all hosted and managed Zetamex Network grids. The name of it is phpSAS (PHP Simple Asset Service), close to the same naming of SRAS (Simple Ruby Asset Service).

What makes it different is that we run it on the latest version of PHP, PHP 7, which is the fastest version of PHP to date. The proxy layer for serving assets and handling the load is NGINX
making it even faster at storing assets and allowing it to serve thousands of requests without causing any major load spikes. Best of all, we are now able to load balance the asset server like never before as assets are served just like a website would be. This means we can put it behind large CDN providers with minimal effort, which is our next project for this.

Resting means Rusting

Constant innovation and planning for the future is nothing new for us, but while we are forging ahead our equipment has always struggled to keep up. In the last year we have drastically changed that and now have plenty of computational resources to test and develop new features with. However, since we are not a 24/7 company, some of that new equipment stays idle for a couple hours each day while we recharge our batteries. As shutting down and then rebooting all these servers and systems would be the best option for the environment, it also means even more time spent not developing.


So the logical conclusion is using the downtime for something useful and so we have setup a Folding@Home machine within our testsystem. Using the spare computational power of the cluster during the night it helps the search for a cure to cancer. You can see the statistics of that in the blogs sidebar, which also has a direct link to the first of the machines we setup this way along with our team number, in case you want to join us. In addition to helping cancer-research the software also allows us to simulate synthetic loads during testruns, which provides us with valuable data towards the stability and reliability of what we test. Finding a cure for cancer one feature at a time if you will. We hope to have sparked some interest in Folding@Home and maybe we will see you on our Folding team in the future.

Patching Vulnerabilities

Making sure everything is up to date and secured properly is our daily bread and butter. Great care is given towards making sure updates are deployed regularly and their behavior is monitored to make sure they do not break anything along the way. It might be a boring and consumes a good deal of time, but the benefits are well worth the effort. Most security holes are patched by these updates, which leaves only a few for us to have an actual look at. What is left is to make sure we actually take advantage of new security measures and implement them into our system.

Unfortunately with security comes inconvenience. Carefully designing security measures as to not impede the ease-of-use of the systems we create is a skill not easily learned. Over time a lot of different approaches to common security issues have sprung up, so there is an array of ones to choose from. Some are highly secure, but very cumbersome to use. Some can be very elegant, but have critical flaws in their design. Our approach is to gradually increase the level of security as to allow our customers and users to adapt and make themselves familiar with the workflow of these measures. This has worked very well in the past, but lately the pace seems to have picked up quite a bit.

Going off the usual blog-writing here to actually address an issue directly. Recently one of our bigger customers has been “attacked” by what can only be described as a “movie-style hacking for leet haxor pros”, ahem. To be more precise, an “attack” was launched towards the user management system, which resulted in a mess of data being entered into one of the databases. The origin of this “attack” was a piece of software distributed by Acunetix , a company specializing in vulnerability testing of websites. According to them, their software has been stolen and is now used without license or consent to stir up all sorts of trouble. While this software offers a great way for website-administrators to test their work and make sure their site is save, it apparently lacks safety of its own. Most companies offering these services make sure that these tests are ran with the consent of the websites owner. Unfortunately their software lacks the capability to make sure this is the case. To us this shows a distinctive lack of care toward security on the web and proper practice of such. What is most disturbing about this, is a lack of transparency and information from their side. Specifically after being promised more information regarding their software and how to deal with the “attacks” it generates.

At this point the damage has been done and we were forced to proceed with implementing additional measures to deal with this sort of thing. We would have liked to evaluate and build a system less intrusive and easier to use. Specifically, we have implemented a Captcha system to prevent automated registrations and injections from automated software. As it is just one click on the Captcha button is not too complex to use yet very effective. We also implemented checks towards making sure registrations come from a legitimate source and use proper credentials. We will continue to increase the security of our systems and, as this case shows, pick up the pace doing so.