10 Years Of Performance

From the perspective of the average user the intricacies of software development may seem akin to magic or some form of science. The reality is often much less glamorous and filled with frustrations. OpenSim as software is now over 10 years old and in that time a lot has changed. From new features, improvements and most importantly bugfixes it has evolved to now support the continuous growth of so many communities. What was once a large team of enthusiasts has largely turned into a handful of individuals still working actively to enhance the software. Naturally development has slowed down and the focus has shifted somewhat to improving the existing features.

While bugs being squashed are an important part of development another is probably even more important. Performance. As the metaverse as a whole grows and the features demand ever more performance to deal with new shiny things such as mesh, whether you are greeted by a slideshow or a fluid movie is ever more important. This especially in times that culture shifts towards demanding engagement in video games and not just pretty pictures.

This is generally an incremental process, but OpenSim is more than the parts it is made up of. As software it relies on a framework to avoid re-inventing the wheel and handle common things without creating even more programming code. As this framework is still in active development it naturally improves as well, bringing new features, performance enhancements and new concepts to the table. The framework underneath is .NET/Mono, which is now entirely owned by Microsoft. Previously the Mono part was under the hands of Xamarin and independent. As project it was aimed to provide a .NET environment for the Linux kernel. Not that long ago Microsoft bought Xamarin however. What you take away from that acquisition is up to debate still, but it has so far brought some advancements to Mono.

OpenSim has somewhat embraced this change and eventually switched to newer versions of the Mono framework as the basis. Mainly driven by the new features and performance of Mono this change has brought many things with it. From the initially bumpy start most of the teething issues have been resolved to the joy of anyone now working with OpenSim. Adjustments to the code and application of new concepts has meant that OpenSim performance has improved significantly. Especially in the active development branch this improvement is quite striking to see. We recently saw this first-hand.

ZetaWorlds, our in-house grid, recently turned 8 years old and to celebrate that occasion a large party was held on one of its regions. This saw nearly 40 people attending the celebrations at some point, which does put a not insignificant load on the various aspects of OpenSim. It was thus rather reassuring to see the performance was not only stable, but in comparison to what it would have been 10 years ago, a lot better than expected. Short of staging an actual test to find the breaking point of it all it did serve to illustrate how far the improvements have come in a decade.

A measure of performance in OpenSim is generally bound to the frames per second the simulator can produce running the region itself. In an ideal scenario this would be 55 frames per second, which serves as the maximum and stable point. Any number lower than this would constitute a situation of backlog, where there are more things to process between each frame than can be reasonably achieve in the allocated time. Situations that produce a lower number can often cascade further essentially grinding everything to a halt. On the side of the actual hardware running OpenSim there is no real measure of performance, instead here we are looking for the amount of resources consumed by the instance of OpenSim. This is the same for any process running on a computer with memory and processor time consumption being the important metrics to monitor.

Not too long ago the general consensus was that each avatar on a region would consume 150-250 megabytes of memory and a good 10% processor load. As resource usage increased the likelihood of Opensim being unable to keep up handling all those resources and thus having to reduce the amount of frames it could produce per second generally happened around 30-40 avatars. This would often mean many gigabytes of memory usage and nearly filling most consumer grade processors of the time. So the following performance improvements are rather striking to see.

It goes without saying that this is an almost ideal situation of all avatars not engaging in a contest of who can throw the most physical objects at each other, which would likely not have had the same results. It is also important to mention that especially when loading a new avatar to a region, much like any significant change, there are times OpenSim chokes on the amount of data to process. This often results in temporary freezes, which resolve themselves, but have a negative impact on the average frame time as is clearly visible.

While there are still ways to go as is evident from the processor usage being pretty high, an event like this being possible and not causing anywhere near the expected resource usage is a massive improvement. From these metrics it seems reasonable to assume that even twice the amount of avatars should, given they are not trying to have a bumper-car session, be within the realm of possibilities.

Much like ZetaWorlds we are looking forward to what the future may hold and hope the quest to improve performance and reduce resource utilization will make even larger events possible in the future. We thank the community ZetaWorlds for making this, in a way unscheduled, performance test possible and obviously for celebrating with us the 8 years of success.

 

PayPal Automatic Payment [Updated]

PayPal is ever evolving, both in the good and bad directions. As you may have noticed, if you have selected automatic payments via PayPal, all these automatic payment plans have been cancelled by PayPal. The information given is sparse, but appears to point at the mismatch of the prorata billing cycle we use to create invoices on the first of the month rather than on the same day the service order is received. PayPal attempts to collect the automatic payments from the first of the month to the date of the order, which results in numerous failed attempts as funds are only released close to the order creation date. We believe PayPal has thus elected to cancel all automatic payments that received too many failures in the past. We have tried to reach out to PayPal for more information as to how to prevent this in the future, but are still awaiting response. For the time being we have disabled automatic payments through PayPal to avoid potential further issues from the failed attempts and to stop the creation of new automatic payments that will be cancelled immediately. We apologize for the inconvenience this may cause, we are working on a solution.

 

[Update 21.10.20]

We have heard back from PayPal with the information that the requests sent to their API contain the correct field for changing the prorata dates, but their API has bundled the calls into a new section. This means we have to wait for our billing system provider to adjust the gateway for PayPal to properly send the data in the correct format. As the changes are fairly recent it seems the update to the gateway is still in progress and the current version of the system does not yet contain the needed changes for prorata to work properly on subscriptions. We will continue to monitor this and have expressed our requirement for this system to work to the provider alongside many others expressing the same need for this change to be accounted for. We expect this to take a few more weeks to be fully resolved and will re-enable subscriptions as soon as we can verify the gateway is working properly for prorata subscriptions. We apologize for the inconvenience.

Service Provider Regulation For Enhanced Security

As an EU company we are bound to abide by the regulations set forward by the EU as implemented by the country of residence. In some cases these regulations are already reflected by local law and can even exceed the requirements set forward by the EU. In the latest round of regulations passed at the beginning of 2020 for enhancing the security of online banking and money transfer in general the EU has set forward a set of regulations that are supposed to make it more difficult to gain unauthorized access to banking accounts through a secondary form of verification.

In reality the implementation of these regulations falls short in a lot of areas and a lot of banking institutions or others handling money transactions are only partially following the regulations set forward. This is despite the clear outlines for what such an implementation requires to provide. Even more concerning is the nature of these implementations often disregarding the specifics of allowing the willful dismissal of participation in the additional verification step. In turn this results in pages and pages worth of complaints against such institutions for both making the steps mandatory despite the optional status and lack of complete adherence to the regulation in general, leaving, in the worst of cases, anyone with the inability to procure or provide verification without access to their funds. The latter being a downright case of discrimination against disabilities in a lot of cases.

We have elected not to follow these regulations, not that we need to as we do not handle sensitive banking information on our end. We do, however, see changes in how our customers pay for their services which are undoubtedly the result of how regulations are implemented with various payment providers we work with. While not strictly our responsibility and far out of our reach to even do something about, we still want to apologize for any inconvenience you may experience paying for your services. We have launched formal complaints with at least one of our payment providers in regards to their failure to properly implement these new EU regulations in a manner that both satisfies the guidelines set forth in the regulations themselves and the implementation abiding the disabilities act.

If you are still having issues with processing your payment with one of our payment processors please do not hesitate to contact us via ticket.

OpenSim Archive – A call for contributions

Zetamex Network is proud to be the sponsor of the OpenSim Archive, a project that aims to centralize a library of resource for and around OpenSimulator for the benefit of creators and those who want to become one. Based around the idea of sharing resources and knowledge, which is one of the core principle behind open source software and the entire FOSS movement, and not re-inventing the wheel for every car that’s made this library aims to aid content creators and everyone else alike. We hope the project will grow and creators will contribute to the enhancement of the worlds we spend our virtual life in. The project is made available to everyone free of charge and even includes things such as old software, compiled OpenSimulator binaries and other useful items. Zetamex Network provides the storage and bandwidth for this project and will handle submissions.

If you are a creator or know of publicly available resource with open licenses we ask for you to get in contact with us to contribute to the library. You can find a contact point on the right, just click the big yellow button or send us an email!

 

Expanding Reach – New Payment Gateways Updated Transaction Fees

The world is a very diverse place with lots of different attitudes to monetary transaction. Philosophical aspects aside the main interest for us as a company in that is the diversity of currencies and ways to turn said currency into goods and services. So in the pursuit of bringing not just more harmony to the world, but also provide easy of access to our services we have added new payment gateways. Additionally we have re-enabled an option on an already existing gateway and upgraded another. Without further ado here they are with the shiny new one leading the front:

Paysafe

Paysafe offers debit-like pins that can be purchased with a certain monetary value attached. These pins can then be used to pay for goods and services online if other payment methods are not available. They have been around for quite some years now and are often the only way for people without access to a bank account to pay for goods and services on the internet. Given they can be purchased with cash they are also quite popular with young people who may just get an allowance handed out in cash. We have partnered with Paysafe to bring this payment method to our brands so that more people can enjoy them.

Stripe SEPA

SEPA is a direct debit system that directly pulls money from the associated bank account. It is quite popular in Europe, because once setup, it is almost hassle-free and so long as the bank account has the appropriate funds on it customers don’t have to worry about their bills not being paid. With the worry of paying bills on time out of the way there is more time to focus on the important things in life.

PayPal Subscriptions

We had previously disabled this functionality due to problems with overcharges and the lack of flexibility in adjusting the subscription when services were changed. These problems have not completely gone away, but recent changes to our billing system have advanced detection of problems in this regard so we feel confident to enable this function once more. With PayPal Subscriptions customers can subscribe to their monthly bill and have it automatically paid without re-authorization. This, much like direct-debit, reduces the complexity of monthly payments for our customers.

 

Transaction Fees

A very important change that has come along with adding new payment gateways is that we are updating our transaction fees. Previously a flat 5% rate was applied to any invoice customers received. This is changing now for a new structure that applies different fees depending on the gateway used. This reflects the contractual conditions we have with the gateway providers and more accurately reflects the additional costs we incur for each gateway. Invoices are automatically updated when a different payment method is selected so customers can see directly which one is better for them. We understand due to many different circumstances some payment methods are less appealing than others so we have tried to reduce the transaction fees where possible. The new transaction fees are as follows:

 

Gateway Fee Gateway Fee Gateway Fee Gateway Fee Gateway Fee
PayPal 5% Stripe 3% Paysafe(new) 15% Stripe SEPA(new) 3% Bank Transfer 7%

 

Additional payment methods may come in the future, if customers have special requirements for payments they can of course contact us in this regard as always. We hope these new payment methods will provide customers with ways to reduce the hassle on monthly bills and provide more alternatives to those without access to the previously available methods.

The Informal Checklist For Grid Owners

Starting your own grid can be a dream, but like any dream it has the potential to turn into a nightmare. We often compare grids with restaurants or other customer-centered businesses, with the reasoning that both share a similar requirement – that a specific culture needs to exist for them to succeed. There are many things to consider and even more things that may not be on everyone’s radar.

It’s a business

“But I am just doing this for fun?!” That may be so and there is inherently nothing wrong with that. The problem comes when you let people sign up with you. Here in Germany, anyone offering signup to a service that is provided to them, free of charge or paid, makes themselves at least partially liable for what goes on within the service provided. That is, until you setup a legal agreement with the user to make them aware of what you are liable for and what you cannot possibly control. Treating a grid as a business, even if you are only running it as a hobby, is vital when you allow people to sign up.

This means first and foremost that a proper Terms of Service agreement and a set of rules for your users to follow is key to keeping some sort of order (and your sanity) intact. It may be that you and most of your users tend to disregard such legal matters, after all what really is in there (apart from lots of legal-speak and the odd joke about monkeys), but when things go wrong this can quickly get dicey. These agreements regulate why, how and who is liable for loss of data, information, breaches of privacy or law. Without it you, the grid owner, are fully liable for it all, because it happened in your “house”. Jurisdictions vary greatly on this, but considering the stricter laws of countries you may gain users from is important, equally are the laws of your own place of dwelling, locations of hardware and software licenses used. Not being in the clear in this department can cost dearly and may even have an impact on your personal life.

Time is money

While you may not value your hobby as much as others whose job it is to keep the world turning, those others may have a different outlook on that. Most pieces of hardware out there doing the dirty work of keeping your grid up and running are associated with monthly costs that need to be handled. Add to this the costs for licenses and software (not to mention actual employees) and you get into a jungle of numbers rather quickly. Having a grasp of what it means to setup a budget and balance costs is important for your success. Managing what costs can be reduced and what needs to ultimately cost more to get the value you need is equally important. This, again, takes extra time, which you may not value as hobby-time, given the very close proximity to doing actual work (if you are accountant that is). This means you end up with actually sacrificing your free time for something you may not be all too keen on handling and we all know what level of enthusiasm we have towards things we don’t like to do and don’t even get the benefit of a monetary return from.

Of course, this is not saying that hiring an accountant and setting up a full business plan is the only way to go, but it is important to keep an overview of what goes on and where funds go. Others make errors, even the big guys, so identifying that someone may overcharge you for something or bills getting lost potentially causing service terminations is quite important. It not only keeps service interruptions from happening, but also keeps the people and companies you work with on your side. Not getting paid is certainly not going to make them want to help you with problems or other support requests should they arise. Valuing others work for what it is, while making sure you are not being ripped off is an important consideration.

Diversity

We have a list of over 400 known grids out there, most of which allow users to sign up with them and thus creating competition. You may not see it as such, but to create a community it needs to be created around something. A gimmick, niche idea or loads of PR are often not enough to keep momentum in that endeavor. Standing out is not easy and requires more than just “this has not been done before”. Even if your aim is not to become the next big thing, having a strategy to build a community is vital if you don’t want to be all alone.

This is one of the big reasons we advise anyone looking to start their own grid to first establish a community before breaking off into the ocean that is the metaverse. Having roots and ties somewhere else can greatly help to bring new users in and create an association that makes it easier for people to find you. It also helps to establish what aim you have for your community, because as a breakaway there is a clear indication wherever you came from and what you may be doing differently that even warrants a closer look.

More than meets the eye

Okay, those titles are getting a bit over the top, but this is kind of an important one. A grid is never just the grid and maybe a website, instead it requires a whole network of additional systems to truly make it work well. This includes, but is not limited to, monitoring systems, backup storage, staff communications, document sharing, test systems, shared calendars, organization tools, wikis and completely custom OpenSim support and management systems. The infrastructure needs of a grid can quickly become a bigger burden of maintenance than the grid itself so selecting the proper systems and software not only provides the necessary force behind a growing grid, but also reduces the overall time you spend dealing with that stuff in the first place. This in turn leaves you more time to actually take care of the grid, its users and their needs and problems that come your way.

Selecting the appropriate things for this is not rocket science, but it’s not trivial either. You may end up falling flat on your face a number of times with stuff that just refuses to work and for that you need alternative plans and solutions. It’s obviously easier to know from the start what to use and what to avoid, which is something we have quite a lot of experience with given how long we have been doing exactly that, but even so there is never a standard solution that fits everyone’s needs equally well. While a lot of solutions offer ways to bend them to specific needs it is not a given property to expect, some stuff just can’t be done in a reasonable manner, and  compromises have to be made, priorities set, and all requirements, present and future, considered. The last thing you want is to be permanently stuck with something that just will not work out as planned.

A Plan B

Not just Primitives can be flexible, most humans are too, though that depends on how much circus blood is in you. Joking aside, keeping a clear head is very important in the day to day operation of a grid. Things change all the time and even the best plans can fall flat when new information or problems come flying your way. Making sure of having some flexibility in your approach will not only help to keep things on track, but will also prevent that level of stress that can spell the disaster that can result from snap decision making. To have a backup plan, both in the sense of planning a secondary option and actually planning how you want to ensure data security, is not an easy task. With the changing nature of the underlying software and the requirements increasing number of users and advances in technology put on the systems that run it all means you are dealing with an exponential increase in the amount of data you handle. Moving that around and making sure it is security and in a state that actually allows recovery in the event something does go belly up may seem like an easy task of just copy and paste, but making a full backup every day is not exactly efficient both in terms of cost of storage and time.

Speaking of security

We all know the fearmongering of your favorite VPN provider, but while their “what ifs” are sometimes over the top they raise similar points one has to consider when dealing with user data. You may have heard about the GDPR and other similar laws such as COPPA. These laws also apply to grids. After all, as previously mentioned, when you let them sign up with you, you become a service provider to them. While both of these laws make specific exceptions for small businesses and hobby projects, complying with them is good practice nonetheless. Unfortunately there is a lot of misinformation and literal interpretations for not just those laws, but most laws governing digital exchanges of data. Thankfully though, when time is taken to read and understand what the aim is, not so much even the literal meaning, but the “good conscience” approach they want you to take, it quickly becomes apparent that complying with these laws is not all that difficult.

It obviously helps to have someone on hand that is well-versed in such matters and deals with them for a living, but not many have a free attorney on speed-dial so learning a bit of legalese and employing a bit of the good old “copy others work” is the minimum. Without diving too deep into the matter, having a firm understanding of what you can and cannot do is still a very helpful tool to have, especially dealing with user concerns and problems. The last thing anyone wants is a fun hobby turning into a court case over some stolen shoes, and yes that has happened.

Data security is also a matter of technological understanding. In short that means setting up hardware and software to ensure your neighbor can walk around in your yard and take things for himself. This is a very time consuming effort as every information security guru you talk to has other ideas of what and what isn’t a secure way of handling cat gifs and social security numbers. It is no less important though. Providing an angle for attack and loss of not just your, but your users’ data, in most jurisdictions makes you liable for the damages. Another reason to properly setup Terms of Service agreements, but just waiving all your responsibility and hoping nothing bad happens is not exactly a strategy most users will want to trust with their data.

The green paper

Money is a complicated subject at the best of times. Having a budget is a good start, sure, but there is more to it than that. Money offers a way to exchange things and services for something that does not require the other party to think of something to return the favor with. It thus creates a big decision to make. Internal money across the grid or external gateways. What gives you the least headaches and what is best for commerce. Even if your aim is not to create a giant shopping mall, with the way money is handled across grids these days users may find themselves wanting support for such a system at their own homes.

Across the spectrum the same question is something you have to ask yourself. When the aim is to break even or even turn the little hobby into something that pays for your next holiday then deciding how to setup the money-service exchange can make or break it all. Handling invoices, rendering payments on time, dealing with credit cards, are quite daunting tasks, and the best approach is not always do-it-yourself. Billing systems are a dime a dozen, but selecting one that works is still not as easy as picking the one that has the nicest logo. By the same token, the way you present all that to your user-turned-customer is equally important.

Fake it until you make it

This is definitely not the policy you should employ when creating an online-presence for your grid or project. While that may work on Instagram it certainly does not work for someone who can tell if someone just slapped their name on something they found looking for “cool website templates”. While there are systems available that come nicely packaged all ready to go, those only provide the basics and to the keen eyed user will make you look like “just another” grid. Taking a bit of time to actually customize and present yourself to your audience can make the difference between registration and a closed tab.

When you have an idea on what you want to convey to the world out there it pays back exponentially to invest some effort into creating a look and feel that is in line with the emotion you want to evoke. That sounds like a lot of “designer” talk and more work as well, but hoping to wow someone in this day in age by creating a logo and adding some buzzwords to the introduction paragraph is not going to work all too well. In the same sense that the world keeps turning, creating a constant flow of information and engagement with your community through something other than throwing parties and posting ads everywhere is going to ensure those who stumble upon your little dwelling are going to stick around for more.

Long term

The vast majority of grids don’t even make it to their first anniversary and not many more stay around for more than 3. Those that manage to maintain an active community usually have the aforementioned diversity and other factors in order alongside a set of dedicated, almost stoic, individuals keeping them up and running. While that is the hard requirement for keeping it all going there are still some softer requirements that apply to create something that lasts. Biggest of all is constant feedback between users and operators, especially in regards to the services you provide to them. What people need and what they get needs to align and grow and evolve along with them. This usually comes naturally, but an unwillingness to let that happen and accept the changing nature can quickly lead to tension.

Equally important for the long term success is having an actual plan on what the long term goal should be. Ideas are easy to come by, but whether they work and catch on is not easily determined. A community may shape this goal with their own wishes and wants, but letting everyone pitch in is not always good for keeping everyone on the same page. Engaging with users over what you want the grid to look like in 5 years and what they may expect to happen is very important. That engagement is going to secure and manifest their feeling of being “at home” and have control over their own destiny. Staying true to your goal, while keeping an open mind to changing the path to it is thus necessary to even make it to the first anniversary party.

On the backend, as mentioned before, constant improvement is also necessary to adjust to changing times. One of the most important aspects in this, long term, is dealing with automation and control. Just a handful of regions and users are easily dealt with, but as things grow they can easily grow out of control. Creating and providing tools to handle certain tasks automatically, giving users the ability to handle their day-to-day business without constant staff interaction or simply making sure bills are paid and servers are up to date is vital. The daily operational workload should ideally stay the same even as the actual workload increases. The only way to achieve this is through automation and clear-cut and standardized procedures, well documented for everyone involved. There is an age-old saying “If you do it thrice, automate” which holds true for most things, because it ends up being a more effective use of your time.

But wait, there is more

So much more. It is difficult to touch on all the topics that end up on the table of a grid owner and there is only so much that can be put in a single article before everyone stops reading and returns to cat videos, so this has to do. Rest assured though, should you elect to kickstart your grid we will be more than happy to assist you all the way from concept to reality. That is, after all, what we do, have done and will keep on doing, that’s our hobby turned business.

There is a contact link somewhere on the right here, while you are there feed the hamster, he has been getting lonely and quite skinny as of late.