Now that the free and open source UltraESB v1.4.0 is out, it is with humbleness that we acknowledge and thank the great open source projects and teams that has brought us where we are!
So if you've ever wondered how we build the worlds best Free and Open Source ESB, in terms of performance, features and ease of use; then read on to learn more!
Apache HttpComponents (Apache License v2.0)
It all started with the Apache HttpComponents project! This is a great project implementing both Java Non-Blocking IO and support for Zero-Copy for HTTP/S, which the UltraESB has been able to fully utilise! Oleg Kalnichevski is the release manager who writes most of the cool code and documents, and is supported by Sebastian, and many others who are always willing to help anyone in a true Apache way! We also use the HttpClient support for NTLM and AWS S3 authentication and the Java-bench code within the SOA ToolBox - now used in the ESB performance benchmark
Spring Framework / Spring Security (Apache License v2.0)
The Spring framework is the basis of our configuration mechanism, and coupled with JDK 6 support for code compilation allowed us to implement a very powerful and limitlessly extensible mediation support, using Java, or any other JSR 223 programming environment such as Javascript, Ruby, Groovy etc.
Spring security is used for HTTP basic and digest authentication, and to tap into any type of user store such as LDAP, databases or hard coded lists.
The Spring framework also powers the JMS transport, and the support for Local or JTA XA transactions of the UltraESB and makes it possible for any transport to process messages in a transactional manner for example SOAP or REST
Apache ZooKeeper (Apache License v2.0)
ZooKeeper is the underlying framework that holds extremely huge clusters of Hadoop together! Its a great framework to work with, and presents a File system like approach to manage complex clustering details. This makes it possible for an UltraESB cluster to be controlled from any node connected to via the UConsole or JMX. I've also been including ZooKeeper as a part of the Distributed Systems module I teach at the University of Colombo, so that both Undergrad and Grad students will have a better appreciation and understanding of this great project.
Jackson JSON Parser (Apache License v2.0) / XML2JSON
Jackson is the fastest JSON parser for Java, and powers the JSON support of the UltraESB, including the Single SQL line REST Data Services with JSON. Jackson also helps in the automatic conversion to/from XML to JSON through the UltraESB which uses the XML2JSON of Paul E. Jones.
PicketBox XACML (LGPL v2.1)
PicketBox is a great XACML engine, and introduced simple and powerful support for fine grained authorisation using XACML. I am yet to write an illustrative sample of this feature, which simplifies using XACML like we simplified using WS-Security an year ago - bypassing performance of WSS4J/Rampart by over 3X. This project was amazingly clean from any JBoss dependencies, and thus has been simple to integrate with the UltraESB in just a couple of days for a POC!
Apache Wink (Apache License v2.0)
Wink is a project still under incubation at Apache, and allows us to write RESTful Web services with JAX-RS to power the front end AJAX UI. Although this is still under incubation, its stable and minimal in size and was ideal for our use.
BeeBole - Pure (MIT License)
Pure is a framework that populates the UConsole UI of HTML5/CSS3 from JSON requests served by Wink. Its a nice library to work with, and a few more examples will make this library a killer.
Apache Shiro (Apache License v2.0)
Shiro secures the UConsole and works great with Wink
JQuery (MIT License) & DataTables (BSD License)
These are two of the best libraries that powers the Ajax UI of the UConsole, and there would be no developer who has not used them in a UI.
Java
Last but not least, Java is the programming language that allowed us to write the UltraESB, and the threading, memory allocation and JMX support, as well as the support for compilation in JDK 6 allowed us to introduce new concepts such as support for mediation fragments in Java, or any JSR 223 scripting language such as Javascript, Ruby, Groovy etc, which other ESBs envies!
VTD XML [Optional end-user downloadable library]
I'd also like to mention an optional library which end users could download themselves and place in the UltraESB library path, that will boost XML processing performance even more. Its VTD XML - licensed under GPL v2 or later. Hence we do not ship it, or link to it directly.
However, once placed along with a binary of VTDFastXML, the UltraESB is able to use the extreme performance of VTD XML to improve its XML performance - especially for XPath evaluation. This was the only area where we had to improve from the ESB performance testing round 4. Just wait for the next round of ESB Performance testing coming up soon, to learn what we mean :)
Friday, May 6, 2011
Thursday, May 5, 2011
UltraESB Code Quality Metrics - updated for v1.4.0
“Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skilful execution; it represents the wise choice of many alternatives.” - William A. Foster
UltraESB v1.4.0 - maintains 43.3% code coverage with samples that double in as unit tests; and 15.5% code comments resulting in a 46% documented API
Read about this ~35MB Open Source Enterprise Service Bus at it at ESB Magic!
UltraESB v1.4.0 - maintains 43.3% code coverage with samples that double in as unit tests; and 15.5% code comments resulting in a 46% documented API
Read about this ~35MB Open Source Enterprise Service Bus at it at ESB Magic!
Tuesday, May 3, 2011
Introducing the UConsole
UConsole is the UltraESB web management console. It is used to monitor the server state, artifacts and lets you mange them as well. It uses a set of bleeding edge technologies to bring the web management of the UltraESB into the table with HTML5 and CSS3 constructs to implement a modern web console with front end processing to render the pages.
UConsole Architecture
The architecture of the UConsole begins from the JMX in the UltraESB back-end where the actual management and monitoring aspects executed via the newly re-factored MXBeans. We have converted from MBeans to MXBeans because of the easiness of management with the MXBeans and the ability to serve complex data as CompositeData objects which can be deconstructed at the management console level easily to provide the management services.
These management services are completely restful and uses the JSON as the media type for the communication. In implementing that we have used Apache Wink deployed on Jetty. So when ever you login to the UltraESB web management console the browser front-end talks to the Wink application serving a set of JSON admin services running on Jetty. These admin services then talks to the UltraESB server via JMX and serves the request from the browser with the content loaded via JMX from the actual UltraESB server.
Now if you look at the browser front-end it is designed to be completely asynchronous and uses AJAX to talk to the Wink JSON services. We mainly use JavaScript with heavy use of jQuery as the key JavaScript framework. However the key piece of code that does the rendering from JSON data received from the Wink application into HTML is PURE and it is really capable of rendering JSON into HTML. At the front end we have been heavily using DataTables to render all the tabular data while using many other jquery.ui extensions to improve the console usability.
The authentication and authorization has been handled by the Apache Shiro framework, which registers as a servlet filer to the Wink RestServlet. That was the uconsole architecture in brief, and the above diagram shows the structure of the console.
UConsole in-action
The management console with the above simple nice architecture gives you a very valuable set of management capabilities for the UltraESB. Assuming that you are running a local UltraESB instance with the console on the same physical machine (which can be started with the uconsole.bat or uconsole.sh scripts available in the bin directory of the UltraESB), it is capable of auto-detecting that node. To manage the locally running node you need to login to the console.
Use the default username and password pair to login to the console which are "admin" and "admin" respectively. If you are managing a remote UltraESB server you need to specify the JMX URL of that UltraESB server on the appropriate input box with the user name and password.
Once you login, you will be able to see a dashboard with different graphs showing the state of the UltraESB server to which you are connected to. There will be 2 more tabs giving you the information on the system and a good set of references for getting started with the UltraESB. One important feature of the UConsole is, it's context sensitive help. If you click on the help icon right next to the heading of any page you will be provided with a nice context sensitive help as shown below;
If we look at the capabilities of this handy console, we better go through the navigation menu of the console to get an understanding of the each and every page it presents.
The first item is the dashboard that we have already talked about and the rest of it can be found in a nice colapsible menu. If you expand that menu, you will be able to see 2 major categories except for the Log viewer, which presents the tail of the log rendered into the console. The 2 major divisions of the UConsole is Node Management and Cluster Management while node management enabled you to manage artifacts on the logged in UltraESB server node, the cluster management section helps you to manage/monitor the complete cluster of the UltraESB from a single console. This comes in very handy when you have a cluster of UltraESBs and you want the central point of management for all the nodes in the cluster.
Under node management, you will get into a tabular list view of all the listed artifact types like, proxy services, sequences, endpoints and so forth. Do not forget to check on the context sensitive help on those pages to get some help as to what each and every page is intended to and what operations you can invoke from those pages. Apart from that, on the top right corner of the detailed artifact pages you will be able to see a control panel and you can use that control panel to invoke the operations for that artifact.
Moving fast to the cluster management, you will only be able to see the submenu section of the cluster management if you have clustering enabled, you may refer to my previous blog or the Clustered deployment of UltraESB article on how to enable clustering and run an UltraESB cluster. This section gives you the complete control over the cluster, where you can navigate through the nodes, to sessions in the cluster and gives a command history view. The most important bit is the control panel and it allows you to issue the cluster controlling commands from the UConsole and monitor those command states, and who has executed that command and so forth.
That is about the usage. Now if you look at the console header banner and the top
right corner you will be able to see the Sign Out an another important feature named Switch Server switch server can be used to switch to another UltraESB node from the one you have been managing. While it can be used to switch to any UltraESB server if you are logged into a node in a given cluster, you will get the liberty of selecting a node in the cluster without typing it in.
So this turned out to be a bit long blog, but still I have many more to share with you all. Anyway you need to have a look at this awesome piece of software and get a feel of it. Download it now to get started with it.
UConsole Architecture
The architecture of the UConsole begins from the JMX in the UltraESB back-end where the actual management and monitoring aspects executed via the newly re-factored MXBeans. We have converted from MBeans to MXBeans because of the easiness of management with the MXBeans and the ability to serve complex data as CompositeData objects which can be deconstructed at the management console level easily to provide the management services.
These management services are completely restful and uses the JSON as the media type for the communication. In implementing that we have used Apache Wink deployed on Jetty. So when ever you login to the UltraESB web management console the browser front-end talks to the Wink application serving a set of JSON admin services running on Jetty. These admin services then talks to the UltraESB server via JMX and serves the request from the browser with the content loaded via JMX from the actual UltraESB server.
Now if you look at the browser front-end it is designed to be completely asynchronous and uses AJAX to talk to the Wink JSON services. We mainly use JavaScript with heavy use of jQuery as the key JavaScript framework. However the key piece of code that does the rendering from JSON data received from the Wink application into HTML is PURE and it is really capable of rendering JSON into HTML. At the front end we have been heavily using DataTables to render all the tabular data while using many other jquery.ui extensions to improve the console usability.
The authentication and authorization has been handled by the Apache Shiro framework, which registers as a servlet filer to the Wink RestServlet. That was the uconsole architecture in brief, and the above diagram shows the structure of the console.
UConsole in-action
The management console with the above simple nice architecture gives you a very valuable set of management capabilities for the UltraESB. Assuming that you are running a local UltraESB instance with the console on the same physical machine (which can be started with the uconsole.bat or uconsole.sh scripts available in the bin directory of the UltraESB), it is capable of auto-detecting that node. To manage the locally running node you need to login to the console.
Use the default username and password pair to login to the console which are "admin" and "admin" respectively. If you are managing a remote UltraESB server you need to specify the JMX URL of that UltraESB server on the appropriate input box with the user name and password.Once you login, you will be able to see a dashboard with different graphs showing the state of the UltraESB server to which you are connected to. There will be 2 more tabs giving you the information on the system and a good set of references for getting started with the UltraESB. One important feature of the UConsole is, it's context sensitive help. If you click on the help icon right next to the heading of any page you will be provided with a nice context sensitive help as shown below;
If we look at the capabilities of this handy console, we better go through the navigation menu of the console to get an understanding of the each and every page it presents.
The first item is the dashboard that we have already talked about and the rest of it can be found in a nice colapsible menu. If you expand that menu, you will be able to see 2 major categories except for the Log viewer, which presents the tail of the log rendered into the console. The 2 major divisions of the UConsole is Node Management and Cluster Management while node management enabled you to manage artifacts on the logged in UltraESB server node, the cluster management section helps you to manage/monitor the complete cluster of the UltraESB from a single console. This comes in very handy when you have a cluster of UltraESBs and you want the central point of management for all the nodes in the cluster.Under node management, you will get into a tabular list view of all the listed artifact types like, proxy services, sequences, endpoints and so forth. Do not forget to check on the context sensitive help on those pages to get some help as to what each and every page is intended to and what operations you can invoke from those pages. Apart from that, on the top right corner of the detailed artifact pages you will be able to see a control panel and you can use that control panel to invoke the operations for that artifact.
Moving fast to the cluster management, you will only be able to see the submenu section of the cluster management if you have clustering enabled, you may refer to my previous blog or the Clustered deployment of UltraESB article on how to enable clustering and run an UltraESB cluster. This section gives you the complete control over the cluster, where you can navigate through the nodes, to sessions in the cluster and gives a command history view. The most important bit is the control panel and it allows you to issue the cluster controlling commands from the UConsole and monitor those command states, and who has executed that command and so forth.
That is about the usage. Now if you look at the console header banner and the top
right corner you will be able to see the Sign Out an another important feature named Switch Server switch server can be used to switch to another UltraESB node from the one you have been managing. While it can be used to switch to any UltraESB server if you are logged into a node in a given cluster, you will get the liberty of selecting a node in the cluster without typing it in.So this turned out to be a bit long blog, but still I have many more to share with you all. Anyway you need to have a look at this awesome piece of software and get a feel of it. Download it now to get started with it.
Monday, May 2, 2011
AdroitLogic announces v1.4.0 of the UltraESB
The latest version brings support for clustering using ZooKeeper, and introduces a web based administration console, and support for Caching, XACML and JSON among many other features.
Singapore - May 2, 2011 - AdroitLogic Private Ltd. announced today the release v1.4.0 of its free and open source Enterprise Service Bus, the UltraESB. The UltraESB is released under the OSI approved GNU Affero General Public License; as well as a zero-dollar non-GPL commercial license which allows unlimited and perpetual use free of charge. First released in January 2010, the UltraESB was the first ESB to utilize memory mapped files and zero-copy, coupled with non-blocking IO to provide extreme levels of performance.
Version v1.4.0 released today offers support for clustering and management of cluster nodes through the new Web based administration console based on JMX. Clustering support is built over Apache ZooKeeper, which is used to manage extremely large clusters of Hadoop nodes. Advanced features such as automatic restart of a complete cluster of nodes - with round-robin processing, or management of a service, or endpoint across a cluster have been made trivial operations. The new version also supports caching, and the current release ships support for ehCache. The new release also ships support for high performance XML processing, utilizing FastXML. This allows end users to integrate third party libraries such as VTD XML with the UltraESB for extremely fast XPath processing, without parsing XML payloads. XACML is supported for fine grained authorization using the PicketBox XACML library, and automatic conversion to/from JSON to XML makes processing JSON payloads even simpler.
In addition to supporting many transports and message formats, the UltraESB also offers support to easily develop JSON based data services with just a single line of configuration, and provides full support for the HTTP transport, including true REST support, basic, digest, NTLM and Amazon S3 authentication using pre-emptive authentication or challenge response mechanisms as applicable. The UltraESB also ships a very high performance WS-Security library that allows it to perform WS-Security operations over 3X faster than solutions based on the WSS4J/Rampart libraries used by other ESBs.
JMX support has been thoroughly improved to utilize MXBeans, and allows detailed monitoring and management of a runtime using JMX consoles such as Zabbix. The UltraESB now ships with a native Zabbix agent for even easier monitoring. As the UltraESB integrates with the Spring framework and supports Java as well as JSR 223 scripting languages, integration with third party libraries and extension is simple and powerful.
UltraESB implements clustering
You may already know from Asankha's blog or from the New item that we have released the 1.4.0 version of the UltraESB few hours ago, and I was waiting and waiting to write about this awesome clustering implementation that I have done for the UltraESB 1.4.0
The story begins with the decision to use ZooKeeper for the clustering implementation, which is what Asankha was initially suggesting me. I've done few feasibility studies and found it to be just right for the clustering implementation that I have been dreaming for years. Why I said I was dreaming is that, most of the ESB implementations out there have not identified the 2 main concerns of clustering;
As explained in the above article UltraESB has cleanly separated the group coordination and cooperative control aspect of the clustering and that has been implemented with using ZooKeeper. On top of this we have written a command framework to command the complete cluster in one go, which enabled us to implement several set of cluster wide management controls.
First and the most important/useful cluster management command that is available on the UltraESB is the Round-Robin graceful restart of the nodes in the cluster. One of the major concerns on the modern ESB clusters was to managing the maintenance restarts without affecting the availability. I haven't come across any ESB out there including commercial once which provides a single control to do the maintenance restart with zero down-time of the service and zero message loss of the system. UltraESB guarantees this with the above operation. Restart being a graceful it guarantees that any node will not restart without serving all the messages it accepted, and the nodes in the cluster are being restarted in a round robin fashion makes sure that the system do not have a state where it doesn't have any live node to dispatch the message to, even if the cluster is just 2 nodes.
One key feature of the new 1.4.0 release is it's web based management console, which I will keep to a separate blog, but I had to mention it here to show you the one of 2 options available for you to try out this in action. So the first option is using the console Cluster Control Panel as shown below;
The other option is to use the jconsole. Which lead me to say a word on the improvements that we have done on the JMX management aspect of the ESB. Now we are using all new MXBeans and not MBeans, and that helped us a lot in improving the management. Will talk about all those on a separate article on the console and its implementation.
Apart from that this command framework enables you to turn on/off sequences, endpoints and proxy services in the complete cluster in one go. Not just that but you could write your own control commands to the UltraESB cluster too and use them in your solution to command the complete cluster.
If you are seeking to try it out the Clustered deployment of UltraESB and control commands is all what you need to go through. You may run the console using the uconsole.sh or uconsole.bat and see this in action in the console too.
That was about group coordination, and then I have evaluated several frameworks for implementing the state replication as that is a significant part of a good clustering implementation and many aspects like scalability and performance measures has to be taken into account when developing such a replication framework. We found the ehCache to be a good fitting distributed cache which can be used to implement state replication, but that used some kind of multi-casting based peer discovery to discover cache peers and we didn't want to rely on that as on cloud it is giving trouble and may not be scalable in a considerably big cluster. There we have used the right amount of salt and use the underlying ZooKeeper based coordination to discover the cache peers of distributed ehCache, thanks to ehCache it had a flexible API and the peer discovery was plugable. I am planning to write a complete article on this aspect of clustering later as the Clustering Part - II and once I am done with it, I will share the link as a comment too.
With that it brings me to the end of the story. Stay tuned to hear more about the UltraESB 1.4.0 and its newly born uconsole.
The story begins with the decision to use ZooKeeper for the clustering implementation, which is what Asankha was initially suggesting me. I've done few feasibility studies and found it to be just right for the clustering implementation that I have been dreaming for years. Why I said I was dreaming is that, most of the ESB implementations out there have not identified the 2 main concerns of clustering;
- Group coordination
- State replication
As explained in the above article UltraESB has cleanly separated the group coordination and cooperative control aspect of the clustering and that has been implemented with using ZooKeeper. On top of this we have written a command framework to command the complete cluster in one go, which enabled us to implement several set of cluster wide management controls.
First and the most important/useful cluster management command that is available on the UltraESB is the Round-Robin graceful restart of the nodes in the cluster. One of the major concerns on the modern ESB clusters was to managing the maintenance restarts without affecting the availability. I haven't come across any ESB out there including commercial once which provides a single control to do the maintenance restart with zero down-time of the service and zero message loss of the system. UltraESB guarantees this with the above operation. Restart being a graceful it guarantees that any node will not restart without serving all the messages it accepted, and the nodes in the cluster are being restarted in a round robin fashion makes sure that the system do not have a state where it doesn't have any live node to dispatch the message to, even if the cluster is just 2 nodes.
One key feature of the new 1.4.0 release is it's web based management console, which I will keep to a separate blog, but I had to mention it here to show you the one of 2 options available for you to try out this in action. So the first option is using the console Cluster Control Panel as shown below;
The other option is to use the jconsole. Which lead me to say a word on the improvements that we have done on the JMX management aspect of the ESB. Now we are using all new MXBeans and not MBeans, and that helped us a lot in improving the management. Will talk about all those on a separate article on the console and its implementation.Apart from that this command framework enables you to turn on/off sequences, endpoints and proxy services in the complete cluster in one go. Not just that but you could write your own control commands to the UltraESB cluster too and use them in your solution to command the complete cluster.
If you are seeking to try it out the Clustered deployment of UltraESB and control commands is all what you need to go through. You may run the console using the uconsole.sh or uconsole.bat and see this in action in the console too.
That was about group coordination, and then I have evaluated several frameworks for implementing the state replication as that is a significant part of a good clustering implementation and many aspects like scalability and performance measures has to be taken into account when developing such a replication framework. We found the ehCache to be a good fitting distributed cache which can be used to implement state replication, but that used some kind of multi-casting based peer discovery to discover cache peers and we didn't want to rely on that as on cloud it is giving trouble and may not be scalable in a considerably big cluster. There we have used the right amount of salt and use the underlying ZooKeeper based coordination to discover the cache peers of distributed ehCache, thanks to ehCache it had a flexible API and the peer discovery was plugable. I am planning to write a complete article on this aspect of clustering later as the Clustering Part - II and once I am done with it, I will share the link as a comment too.
With that it brings me to the end of the story. Stay tuned to hear more about the UltraESB 1.4.0 and its newly born uconsole.
Subscribe to:
Posts (Atom)