As some of you might know, we started our SmarterMail 17.x BETA in March. We've paused BETA releases for the last six weeks or so, however, and want to explain why. We definitely hit some roadblocks, which changed our direction, but the end result is very positive!
We're in a much better position, now, and things are progressing quickly. Below, we'll discuss some of what we've learned, some of the changs you can expect, and then, finally, we'll lay out the roadmap for the next couple of BETA releases. As it is, we're anticipating the full public release of SmarterMail 17.x very soon.
The initial plan for SmarterMail 17.x was to release BETAs in phases, with each phase focusing on a few major technology changes. This would give people the ability to understand what was included in each Phase and allow the development team to fix Bugs in isolated areas as customers found them.
One of the major Phases was to build a Universal Data Layer that all SmarterMail file/data access would flow through. It would allow us to dramatically simplify the complexity of code and provide one caching layer across the entire product. This would reduce memory consumption while making it much simpler to integrate different datastores such as SQLite, MySQL or SQL Server.
Over the years we have been very vocal on how SmarterMail is one of the most efficient and highest performing mail servers on the market. We intentionally avoided database engines more than a decade ago because they were unable to achieve the results that ISP's, hosting companies and enterprises required. Many of our competitors at the time used a variety of databases and is why they had so many issues. A good example of this was, and still IS, Microsoft Exchange. However, after decades of improvements in all database engines and some initial testing, we thought the time had come to introduce it into SmarterMail 17.x.
Unfortunately, we discovered a significant decrease in performance when making this change. All of the database engines we tested were unable to reach the speeds that were seen in previous SmarterMail releases. As you can imagine, this was somewhat disappointing and far from our original goal of increasing the speed of data access and lowering overall disk i/o in SmarterMail 17.x. It also clearly impacted our BETA plans and release date.
But, a lot of good has come from it.
The Universal Data Layer, which also includes a single caching layer, has been incredibly helpful in transitioning back to a file-based datastores similar to what we have had in the past. For example, instead of XML files, mailing lists utilizing SQLite, and some other methods of storing data, we have standardized with JSON across all of SmarterMail. (Except for mailbox storage, which is going to stay in our extremely efficient .GRP files.) All files and data have been refactored as well, allowing easier access to higher demand data, and the single caching layer across the entire system has reduced memory and improved performance. With these changes, SmarterMail 17.x is seeing a 20% decrease in disk i/o which is absolutely incredible since we have even more confirmation on how high performing previous versions of SmarterMail were.
Both Linux and Clustering are still being taken into account with our file based datastores and will yield significant benefits based on our testing. Both of these will be coming in versions after SmarterMail 17.x.
Face-to-Face With Microsoft
A few weeks ago, we sent a number of the SmarterMail developers on a trip to Redmond, home of Microsoft. This trip was an indepth week into various Microsoft protocols. SmarterTools developers worked with some of the original developers of MAPI, EWS, EAS and more. While we've exchanged emails with the teams over the years, it's always nice to be able to put a “face to a name”, and real time, in-person discussions always seem to yield more results than email chains. Interestingly, we also found out how rare we are in that we are one of a few companies who have ever done a server side implementation of MAPI. We did know most of our competitors created plugins for Outlook to simulate MAPI but its not MAPI and doesn't have all the functionality of MAPI.
As you can imagine, a LOT came out of those discussions and meetings. In SmarterMail 17.x, customers are going to see significant improvements in all protocols such as EAS and EWS, and MAPI will be a very complete implementation equalling that of Exchange.
Another interesting tidbit was mentioned: it turns out Outlook for the desktop was never meant to support EAS. Some people we talked to said that while it DOES work, it was never meant to nor will Microsoft support issues with EAS and Outlook on the desktop. EAS remains the standard for mobile syncing, but for Outlook on the desktop their suggestion is to use a different sync method or use Windows mail.
Next BETA Releases
We will restart the BETA releases of SmarterMail 17.x the week of July 30th and it will include the Universal Data Layer and Caching while converting all files to the new file datastores. We will spend a couple of weeks working with customers to ensure all conversions are working properly and eliminate any bugs that are found. Soon after, we will be releasing a BETA that includes MAPI and go through the same process.
Between SmarterMail 16.x, which introduced the new interface, and SmarterMail 17.x with an entirely new file system and file structure, SmarterMail has been completely rewritten. We're using the latest technologies and have consolidated all the features and functionality added over the last sixteen years into one common code base. All of these changes mean we have less code, cleaner code, better CPU utilization, less disk i/o and less memory usage than any time in the history of the product. This helps users and system administrators, of course, but it also places us in a great place for future product development and new features. More is on the horizon, so stay tuned.
One Final Note
As with ALL BETAs, we do not recommend pushing any BETA build to a full production server. This is especially true for SmarterMail 17.x. As noted there are EXTENSIVE back end processing and configuration file changes that will absolutely and unequivocally prevent you from rolling back to a previous version of SmarterMail. Therefore: DO NOT put a BETA build of SmarterMail 17.x on any production mail server. In addition, if you're upgrading a 16.x installation to test 17.x features -- not in a production environment, of course -- after you perform the upgrade you'll want to let SmarterMail run so it can work through all domains and users and complete the upgrade process. This will take a little more CPU and disk as it does this. Once it's all done, make a backup of the 17.x installation. Due to all of the back end changes, you cannot restore SmarterMail 16.x files to a 17.x server.
Getting Started with the BETA
Once you've received your BETA key, visit the SmarterMail 17.x BETA category in our Community! There, you can communicate with other testers (as well as the SmarterTools development team), view release notes for the most recent BETA releases and stay up-to-date with any additional BETA news and announcements.
This post was originally titled "SmarterMail 17 Coming in September. As it's now October, and we're still in BETA, we wanted to provide an update.
One of the problems with promising release dates is just this: things slip. Issues can arise, or, and we run into this occasionally, we add in a major feature and then want to keep adding to that feature to make it better. In the section "Face-to-Face With Microsoft", we discussed the benefits and knowledge we received when spending a week up in Redmond with the Microsoft protocols teams. Coming back from that, we had an entirely new way of looking at how we interface with EAS, EWS, CalDAV and MAPI. That forced us to modify the way we were handling those protocols, as well as modify SmarterMail's back end and further modify its file system. As you can imagine, this takes time.
As it stands, we're in the final stages of prepping the next BETA of SmarterMail 17.x, which should be out in the next week or two. Then, pending the results of those tests and the feedback we get, we'll have the final, public release.
When will THAT be? Well, that's hard to say. We have a LOT of exciting changes on the horizon: we have major releases of SmarterMail and SmarterTrack coming up, as well as some additional, non product-related changes that we feel are going to make customers and partners VERY happy. The plan is to release them all at the same time. As we've said, putting a strict timeline on these items is difficult, because issues always arise. The simple answer for "When" is: when they're all done and ready, which will be (well) before the end of the year. That's about as exact as we'd like to be at this point. Just stay tuned: it will all be worth the wait!