UltraESB v1.6.2 released!

AdroitLogic Private Ltd. announced today the release v1.6.2 of its free and open source Enterprise Service Bus, the UltraESB. The UltraESB is released under the OSI approved GNU Affero General Public License; as well as a zero-dollar non-GPL commercial license which allows unlimited and perpetual production use free of charge. First released in January 2010, the UltraESB was the first ESB to utilize memory mapped files and zero-copy, coupled with non-blocking IO to provide extreme levels of performance.

Version v1.6.2 released today is mainly a maintenance release with many enhancements to the JMS transport, including fixing the ability to implement store-n-forward MEP with message re-delivery. The 1.6.0 release has been battle tested and hardened to become the 1.6.1 and then 1.6.2 with more stability. It also has few enhancements to the clustering support, REST support, file cache handling, enhanced logging and the server startup.

This minor release contains a much enhanced logging where all the error level logs have been re-visited to confirm whether they are actual errors from a users point of view. This enables users to setup alerting based on the error log statements using a third party log analyzing tools. The enhancements to the logs make sure that an error log is an actual error which requires a user attention and hence should result in an alert. Further on the startup if the server detects a compilation failure of any sequences or configuration errors the server will automatically be shutdown gracefully without accepting any messages, and the server state will be set to FAILED so that administrator can inspect improved logs regarding the error and fix those to start the server cleanly next time.

The ZooKeeper based clustering support has also been improved, where at a sudden forced shutdown will not affect the next start of the server, by acquiring the previous ZooKeeper session and terminating it properly to maintain the server session details properly, etc.. This release also contains a fix to the windows startup bat scripts for an issue with starting samples on the windows environment.

The v1.6.2 release is approximately ~35MB in size and includes over 70 samples, test utilities and documentation etc to get started quickly. The release can be downloaded from http://adroitlogic.org/download

UltraESB adds support for the routing and mediation based on Protocol Buffers

UltraESB adds support for the routing and mediatio...: Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages – Java, C++, or Python.

UltraESB is the best ESB in usability, performance and extensibility. As many organizations are now moving to protocol buffers to take advantage of its performance and multiple language support, it has been the right time that we introduced support for routing and mediating such messages based on protocol buffers.

Read full story

Past, Present and Future of ...: UltraESB Documenation

AdroitLogic has released the first phase of the official documentation for the UltraESB. This documentation is a complete reference of the UltraESB product and the associated tools.

Does your ESB vendor charge you for access to Premium Content? Well, all UltraESB documentation, including our premium Production Support guides are free!

ESB Performance Comparison Results and Details published

Last week we published the results of the Round #5 of ESB Performance Testing, and we've already had feedback to improve the configurations for some of the ESBs.

Of-course, we would be happy to work with other ESB vendors to ensure that the ESB configuration resources now hosted publicly at https://bitbucket.org/adroitlogic/esbperformance is optimal for each ESB compared in both configuration and performance tuning details.

We've just published detailed instructions for re-executing the Round #5 on an EC2 node - read about it here on http://esbperformance.org

ESB Performance Testing.. Getting ready for the next round

Its been sometime since we've compared the performance of free and open source ESBs.. and now its time for the next round..

Amazon EC2 AMI to be published!
At the end of this round, we will publish an Amazon EC2 machine image (AMI) with the selected ESBs so that any end-user can simply fire up an instance of this image, and re-run the performance suite to compare the ESBs of his choice!

Read all about it here on ESB Magic and at http://esbperformance.org  

AdroitLogic announces v1.5.0 of the UltraESB

The latest version brings support for Command Line Interface for management, improved JMX support, and integration with monitoring solutions and a high performance RAM disk based file cache

Singapore - May 20, 2011 - AdroitLogic Private Ltd. announced today the release v1.5.0 of its free and open source Enterprise Service Bus, the UltraESB. The UltraESB is released under the OSI approved GNU Affero General Public License; as well as a zero-dollar non-GPL commercial license which allows unlimited and perpetual use free of charge. First released in January 2010, the UltraESB was the first ESB to utilize memory mapped files and zero-copy, coupled with non-blocking IO to provide extreme levels of performance.

Version v1.5.0 released today offers support for a new Command Line Interface (CLI) for management allowing interactive and scripted management preferred by system administrators. The new version also has more attributes exposed for JMX based management and monitoring, including enhancements to the UConsole - web based management console. A RAM disk based file cache improves performance much further, and the exact numbers will be announced shortly with the complete ESB Performance Test Suite (http://esbperformance.org) used previously by many vendors to allow end users to measure the performance and compare it with any other open source or commercial ESB of choice.

The new release also includes a facility to automatically register ESB elements into the popular Zabbix - Open Source Monitoring solution. The close integration allows users to monitor the ESB, set triggers when certain thresholds are reached or SLAs are violated etc, and generate graphs for activity across weeks, months or even years. Improved connection debugging allows the generation of detailed connection information on HTTP level failures so that problems with clients or backend services etc can be quickly understood. The UConsole also allows the user to view the tail end of the logfile, and change the underlying Log4J configuration at runtime, and now supports the IE browser as well and includes secured access with JMX

The v1.5.0 release is approximately ~35MB in size and includes over 70 samples, test utilities and documentation etc to get started quickly. The release can be downloaded from http://adroitlogic.org

Monitoring UltraESB with Zabbix

UltraESB is an open source project geared towards developing the worlds best ESB! It is already the best in performance, ease of use, and many features, but we are keen to further improve its core features for mediation, management and monitoring!

While working on the management/monitoring aspects of the UltraESB we have been introduced to Zabbix by one of our users. We already had a good web based management console called the UConsole, and a JSON REST services API for managing and monitoring, with a command line client to be released in the near future. Even though there will be many possibilities for management, each of these uses the same JMX infrastructure underneath. After all, JMX is supposed to be the management framework for all Java based applications.

Now, coming back to the topic, Zabbix monitoring has also been integrated with the UltraESB via the JMX API, facilitated by a native Zabbix agent that ships with the UltraESB v1.4.x onwards. The latest article is on the basics of monitoring the UltraESB using Zabbix.


Sometime back we mentioned about a set of great open source projects that the UltraESB utilizes, and we will soon be listing the free and open source Zabbix monitoring solution there! This again emphasizes the goal of Adroitlogic which is to build the best ESB out there, and not to re-implement all the existing wheels in the process! We will always select the best open source projects/tools out there which is not our core competency, and say publicly that they are what we use. Of course we do it without re-branding them as our own components the way some vendors sell others' open source projects as new products these days :)

Thanking great Open Source projects and teams!

Now that the free and open source UltraESB v1.4.0 is out, it is with humbleness that we acknowledge and thank the great open source projects and teams that has brought us where we are!

So if you've ever wondered how we build the worlds best Free and Open Source ESB, in terms of performance, features and ease of use; then read on to learn more!

Apache HttpComponents (Apache License v2.0)
It all started with the Apache HttpComponents project! This is a great project implementing both Java Non-Blocking IO and support for Zero-Copy for HTTP/S, which the UltraESB has been able to fully utilise! Oleg Kalnichevski is the release manager who writes most of the cool code and documents, and is supported by Sebastian, and many others who are always willing to help anyone in a true Apache way! We also use the HttpClient support for NTLM and AWS S3 authentication and the Java-bench code within the SOA ToolBox - now used in the ESB performance benchmark

Spring Framework / Spring Security (Apache License v2.0)
The Spring framework is the basis of our configuration mechanism, and coupled with JDK 6 support for code compilation allowed us to implement a very powerful and limitlessly extensible mediation support, using Java, or any other JSR 223 programming environment such as Javascript, Ruby, Groovy etc.
Spring security is used for HTTP basic and digest authentication, and to tap into any type of user store such as LDAP, databases or hard coded lists.

The Spring framework also powers the JMS transport, and the support for Local or JTA XA transactions of the UltraESB and makes it possible for any transport to process messages in a transactional manner for example SOAP or REST

Apache ZooKeeper (Apache License v2.0)
ZooKeeper is the underlying framework that holds extremely huge clusters of Hadoop together! Its a great framework to work with, and presents a File system like approach to manage complex clustering details. This makes it possible for an UltraESB cluster to be controlled from any node connected to via the UConsole or JMX. I've also been including ZooKeeper as a part of the Distributed Systems module I teach at the University of Colombo, so that both Undergrad and Grad students will have a better appreciation and understanding of this great project.

Jackson JSON Parser (Apache License v2.0) / XML2JSON
Jackson is the fastest JSON parser for Java, and powers the JSON support of the UltraESB, including the Single SQL line REST Data Services with JSON. Jackson also helps in the automatic conversion to/from XML to JSON through the UltraESB which uses the XML2JSON of Paul E. Jones.

PicketBox XACML (LGPL v2.1)
PicketBox is a great XACML engine, and introduced simple and powerful support for fine grained authorisation using XACML. I am yet to write an illustrative sample of this feature, which simplifies using XACML like we simplified using WS-Security an year ago - bypassing performance of WSS4J/Rampart by over 3X. This project was amazingly clean from any JBoss dependencies, and thus has been simple to integrate with the UltraESB in just a couple of days for a POC!

Apache Wink (Apache License v2.0)
Wink is a project still under incubation at Apache, and allows us to write RESTful Web services with JAX-RS to power the front end AJAX UI. Although this is still under incubation, its stable and minimal in size and was ideal for our use.

BeeBole - Pure (MIT License)
Pure is a framework that populates the UConsole UI of HTML5/CSS3 from JSON requests served by Wink. Its a nice library to work with, and a few more examples will make this library a killer.

Apache Shiro (Apache License v2.0)
Shiro secures the UConsole and works great with Wink

JQuery (MIT License) & DataTables (BSD License)

These are two of the best libraries that powers the Ajax UI of the UConsole, and there would be no developer who has not used them in a UI.


Java
Last but not least, Java is the programming language that allowed us to write the UltraESB, and the threading, memory allocation and JMX support, as well as the support for compilation in JDK 6 allowed us to introduce new concepts such as support for mediation fragments in Java, or any JSR 223 scripting language such as Javascript, Ruby, Groovy etc, which other ESBs envies!


VTD XML [Optional end-user downloadable library]
I'd also like to mention an optional library which end users could download themselves and place in the UltraESB library path, that will boost XML processing performance even more. Its VTD XML - licensed under GPL v2 or later. Hence we do not ship it, or link to it directly.


However, once placed along with a binary of VTDFastXML, the UltraESB is able to use the extreme performance of VTD XML to improve its XML performance - especially for XPath evaluation. This was the only area where we had to improve from the ESB performance testing round 4. Just wait for the next round of ESB Performance testing coming up soon, to learn what we mean :)

UltraESB Code Quality Metrics - updated for v1.4.0

“Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skilful execution; it represents the wise choice of many alternatives.” - William A. Foster

UltraESB v1.4.0 - maintains 43.3% code coverage with samples that double in as unit tests; and 15.5% code comments resulting in a 46% documented API

Read about this ~35MB Open Source Enterprise Service Bus at it at ESB Magic!

Introducing the UConsole

UConsole is the UltraESB web management console. It is used to monitor the server state, artifacts and lets you mange them as well. It uses a set of bleeding edge technologies to bring the web management of the UltraESB into the table with HTML5 and CSS3 constructs to implement a modern web console with front end processing to render the pages.

UConsole Architecture

The architecture of the UConsole begins from the JMX in the UltraESB back-end where the actual management and monitoring aspects executed via the newly re-factored MXBeans. We have converted from MBeans to MXBeans because of the easiness of management with the MXBeans and the ability to serve complex data as CompositeData objects which can be deconstructed at the management console level easily to provide the management services.


These management services are completely restful and uses the JSON as the media type for the communication. In implementing that we have used Apache Wink deployed on Jetty. So when ever you login to the UltraESB web management console the browser front-end talks to the Wink application serving a set of JSON admin services running on Jetty. These admin services then talks to the UltraESB server via JMX and serves the request from the browser with the content loaded via JMX from the actual UltraESB server.

Now if you look at the browser front-end it is designed to be completely asynchronous and uses AJAX to talk to the Wink JSON services. We mainly use JavaScript with heavy use of jQuery as the key JavaScript framework. However the key piece of code that does the rendering from JSON data received from the Wink application into HTML is PURE and it is really capable of rendering JSON into HTML. At the front end we have been heavily using DataTables to render all the tabular data while using many other jquery.ui extensions to improve the console usability.

The authentication and authorization has been handled by the Apache Shiro framework, which registers as a servlet filer to the Wink RestServlet. That was the uconsole architecture in brief, and the above diagram shows the structure of the console.

UConsole in-action

The management console with the above simple nice architecture gives you a very valuable set of management capabilities for the UltraESB. Assuming that you are running a local UltraESB instance with the console on the same physical machine (which can be started with the uconsole.bat or uconsole.sh scripts available in the bin directory of the UltraESB), it is capable of auto-detecting that node. To manage the locally running node you need to login to the console.
Use the default username and password pair to login to the console which are "admin" and "admin" respectively. If you are managing a remote UltraESB server you need to specify the JMX URL of that UltraESB server on the appropriate input box with the user name and password.

Once you login, you will be able to see a dashboard with different graphs showing the state of the UltraESB server to which you are connected to. There will be 2 more tabs giving you the information on the system and a good set of references for getting started with the UltraESB. One important feature of the UConsole is, it's context sensitive help. If you click on the help icon right next to the heading of any page you will be provided with a nice context sensitive help as shown below;


If we look at the capabilities of this handy console, we better go through the navigation menu of the console to get an understanding of the each and every page it presents.

The first item is the dashboard that we have already talked about and the rest of it can be found in a nice colapsible menu. If you expand that menu, you will be able to see 2 major categories except for the Log viewer, which presents the tail of the log rendered into the console. The 2 major divisions of the UConsole is Node Management and Cluster Management while node management enabled you to manage artifacts on the logged in UltraESB server node, the cluster management section helps you to manage/monitor the complete cluster of the UltraESB from a single console. This comes in very handy when you have a cluster of UltraESBs and you want the central point of management for all the nodes in the cluster.

Under node management, you will get into a tabular list view of all the listed artifact types like, proxy services, sequences, endpoints and so forth. Do not forget to check on the context sensitive help on those pages to get some help as to what each and every page is intended to and what operations you can invoke from those pages. Apart from that, on the top right corner of the detailed artifact pages you will be able to see a control panel and you can use that control panel to invoke the operations for that artifact.

Moving fast to the cluster management, you will only be able to see the submenu section of the cluster management if you have clustering enabled, you may refer to my previous blog or the Clustered deployment of UltraESB article on how to enable clustering and run an UltraESB cluster. This section gives you the complete control over the cluster, where you can navigate through the nodes, to sessions in the cluster and gives a command history view. The most important bit is the control panel and it allows you to issue the cluster controlling commands from the UConsole and monitor those command states, and who has executed that command and so forth.

That is about the usage. Now if you look at the console header banner and the top right corner you will be able to see the Sign Out an another important feature named Switch Server switch server can be used to switch to another UltraESB node from the one you have been managing. While it can be used to switch to any UltraESB server if you are logged into a node in a given cluster, you will get the liberty of selecting a node in the cluster without typing it in.

So this turned out to be a bit long blog, but still I have many more to share with you all. Anyway you need to have a look at this awesome piece of software and get a feel of it. Download it now to get started with it.

AdroitLogic announces v1.4.0 of the UltraESB

The latest version brings support for clustering using ZooKeeper, and introduces a web based administration console, and support for Caching, XACML and JSON among many other features.

Singapore - May 2, 2011 - AdroitLogic Private Ltd. announced today the release v1.4.0 of its free and open source Enterprise Service Bus, the UltraESB. The UltraESB is released under the OSI approved GNU Affero General Public License; as well as a zero-dollar non-GPL commercial license which allows unlimited and perpetual use free of charge. First released in January 2010, the UltraESB was the first ESB to utilize memory mapped files and zero-copy, coupled with non-blocking IO to provide extreme levels of performance.

Version v1.4.0 released today offers support for clustering and management of cluster nodes through the new Web based administration console based on JMX. Clustering support is built over Apache ZooKeeper, which is used to manage extremely large clusters of Hadoop nodes. Advanced features such as automatic restart of a complete cluster of nodes - with round-robin processing, or management of a service, or endpoint across a cluster have been made trivial operations. The new version also supports caching, and the current release ships support for ehCache. The new release also ships support for high performance XML processing, utilizing FastXML. This allows end users to integrate third party libraries such as VTD XML with the UltraESB for extremely fast XPath processing, without parsing XML payloads. XACML is supported for fine grained authorization using the PicketBox  XACML library, and automatic conversion to/from JSON to XML makes processing JSON payloads even simpler.

In addition to supporting many transports and message formats, the UltraESB also offers support to easily develop JSON based data services with just a single line of configuration, and provides full support for the HTTP transport, including true REST support, basic, digest, NTLM and Amazon S3 authentication using pre-emptive authentication or challenge response mechanisms as applicable. The UltraESB also ships a very high performance WS-Security library that allows it to perform WS-Security operations over 3X faster than solutions based on the WSS4J/Rampart libraries used by other ESBs.

JMX support has been thoroughly improved to utilize MXBeans, and allows detailed monitoring and management of a runtime using JMX consoles such as Zabbix. The UltraESB now ships with a native Zabbix agent for even easier monitoring. As the UltraESB integrates with the Spring framework and supports Java as well as JSR 223 scripting languages, integration with third party libraries and extension is simple and powerful.

UltraESB implements clustering

You may already know from Asankha's blog or from the New item that we have released the 1.4.0 version of the UltraESB few hours ago, and I was waiting and waiting to write about this awesome clustering implementation that I have done for the UltraESB 1.4.0

The story begins with the decision to use ZooKeeper for the clustering implementation, which is what Asankha was initially suggesting me. I've done few feasibility studies and found it to be just right for the clustering implementation that I have been dreaming for years. Why I said I was dreaming is that, most of the ESB implementations out there have not identified the 2 main concerns of clustering;

• Group coordination


• State replication

to be different and they failed to keep those 2 concerns nicely separated. What I believe is that these 2 concerns of clustering needs to be managed separately and hence implemented with right amount of isolation yet integrating them together to work nicely. It is like spice or salt for food, you need that but only the just right amount, too less or too much gives you trouble. I have explained about this in more detail in my Clustering Part I article.

As explained in the above article UltraESB has cleanly separated the group coordination and cooperative control aspect of the clustering and that has been implemented with using ZooKeeper. On top of this we have written a command framework to command the complete cluster in one go, which enabled us to implement several set of cluster wide management controls.

First and the most important/useful cluster management command that is available on the UltraESB is the Round-Robin graceful restart of the nodes in the cluster. One of the major concerns on the modern ESB clusters was to managing the maintenance restarts without affecting the availability. I haven't come across any ESB out there including commercial once which provides a single control to do the maintenance restart with zero down-time of the service and zero message loss of the system. UltraESB guarantees this with the above operation. Restart being a graceful it guarantees that any node will not restart without serving all the messages it accepted, and the nodes in the cluster are being restarted in a round robin fashion makes sure that the system do not have a state where it doesn't have any live node to dispatch the message to, even if the cluster is just 2 nodes.

One key feature of the new 1.4.0 release is it's web based management console, which I will keep to a separate blog, but I had to mention it here to show you the one of 2 options available for you to try out this in action. So the first option is using the console Cluster Control Panel as shown below;

The other option is to use the jconsole. Which lead me to say a word on the improvements that we have done on the JMX management aspect of the ESB. Now we are using all new MXBeans and not MBeans, and that helped us a lot in improving the management. Will talk about all those on a separate article on the console and its implementation.

Apart from that this command framework enables you to turn on/off sequences, endpoints and proxy services in the complete cluster in one go. Not just that but you could write your own control commands to the UltraESB cluster too and use them in your solution to command the complete cluster.

If you are seeking to try it out the Clustered deployment of UltraESB and control commands is all what you need to go through. You may run the console using the uconsole.sh or uconsole.bat and see this in action in the console too.

That was about group coordination, and then I have evaluated several frameworks for implementing the state replication as that is a significant part of a good clustering implementation and many aspects like scalability and performance measures has to be taken into account when developing such a replication framework. We found the ehCache to be a good fitting distributed cache which can be used to implement state replication, but that used some kind of multi-casting based peer discovery to discover cache peers and we didn't want to rely on that as on cloud it is giving trouble and may not be scalable in a considerably big cluster. There we have used the right amount of salt and use the underlying ZooKeeper based coordination to discover the cache peers of distributed ehCache, thanks to ehCache it had a flexible API and the peer discovery was plugable. I am planning to write a complete article on this aspect of clustering later as the Clustering Part - II and once I am done with it, I will share the link as a comment too.

With that it brings me to the end of the story. Stay tuned to hear more about the UltraESB 1.4.0 and its newly born uconsole.

UltraESB v1.4.0 development builds are available for feedback!

Good cooking takes time. If you are made to wait, it is to serve you better, and to please you.

- MENU OF RESTAURANT ANTOINE. NEW ORLEANS

Anyone who has read the Mythical Man Month by Brooks will remember the famous statement on the menu of Restaurant Antoine. Likewise, we've taken a bit of time, to do a great release of the UltraESB v1.4.0, to please you better! The release is expected later this month, but we've started releasing public development builds for user feedback.

Some of the highlights of the new release are:

• Support for clustered deployment and management of a cluster (based on Apache ZooKeeper)
• Configuration updates, includes and removals at runtime including a "prepare" phase that detects possible issues before a configuration switch
• An extremely lightweight Web based console for management of the UltraESB and basic monitoring
• Complete revamp of JMX functionality to use MXBeans, and including a built-in Zabbix monitoring agent
• Support for local and distributed caching via ehCache (expect support for memcached soon!)
• XACML support using PicketBox XACML
• Support for save/restore of test setup from the ToolBox
• Many more enhancements and rectification of issues

Here is a brief screenshot from the web console

And here is a view of a proxy service shutdown command on a cluster, where one can see the list of nodes where the command completed successfully, or failed

The console also allows one to quickly load new configuration fragments, or unload or update loaded fragments

• You can download the latest binary from: http://downloads.adroitlogic.com/temp/adroitlogic-ultraesb-commercial-1.4.0-bin.zip

• To checkout the source code, use Mercurial as follows:hg clone https://bitbucket.org/adroitlogic/ultraesb

• To download a source Zip, or to raise an issue, please visit https://bitbucket.org/adroitlogic/ultraesb

• For your feedback on the new features, please use http://getsatisfaction.com/adroitlogic

Note: Some of the pages/parts of the pages of the Web console are still under construction