Comparison of Java APIs for Excel

The Aspose folks have put up a comparison of their Java Excel API vs popular open source APIs such as JExcelAPI and Apache POI.

The comparison is worth a persual:
http://www.aspose.com/docs/display/cellsjava/Aspose.Cells+for+Java+Vs+Open+Source+Competitors

Using Spring MVC to create REST style services

We have been always using Apache CXF or Axis 2 to build web services, but recently I was pretty impressed with the simplicity with which we can create REST style services using Spring MVC. OOTB, Spring MVC integrates with Jackson (for JSON serialization) and JAXB (for XML serialization).

We just have to annotate our Spring Controller class with @RequestMapping and @ResponseBody attributes and let Spring handle the rest. The following links would provide more info on the same.

http://www.javacodegeeks.com/2013/04/spring-mvc-easy-rest-based-json-services-with-responsebody.html

http://xpadro.blogspot.in/2013/02/create-and-test-rest-services-with.html

Using Spring MVC with Ext-JS MVC
http://justinrodenbostel.wordpress.com/2013/03/15/restful-spring-mvc-and-extjs-episode-1-the-spring-stuff/
http://justinrodenbostel.wordpress.com/2013/04/02/restful-spring-mvc-and-extjs-episode-2-the-extjs-stuff/

Maths Formula Evaluators

A team mate today introduced me to a couple of open-source libraries that can be used for evaluating mathametical functions as strings. One can set variables and evaluate the expression.

http://www.formula4j.com/
http://www.japisoft.com/formula/index.html

Are you a "chicken" or "pig" in a project?

Loved the below classic story of the Chicken and Pig. In any project you would have committed people and folks who only get involved :)

http://www.implementingscrum.com/2006/09/11/the-classic-story-of-the-pig-and-chicken/

Ruminating on MongoDB concurrency

In my previous post, I had discussed about the transaction support in MongoDB. Since MongoDB implements a ReadersWriter lock at the database level, I was a bit concerned that a writer lock for a long query may block all read operations for the time the query is running.

For e.g. If we fire a mongoDB query that updates 10,000 documents that would take 5 mins. Would all reads be blocked for the 5 mins till all the records are udpated? If yes, then that would be disastrous.

Fortunately this does not happen, due to lock yielding as stated on the MongoDB site.
Snippet from the site:
"Write operations that affect multiple documents (i.e. update() with the multi parameter,) will yield periodically to allow read operations during these long write operations. Similarly, long running read locks will yield periodically to ensure that write operations have the opportunity to complete."

Read locks are shared, but "read locks" block write locks from being acquired. Write locks prevent both other writes and reads. But MongoDB operations yield periodically to keep other threads waiting for locks from starving. An interesting blog post that shows stats on the performance of MongoDB locks is available here. 

Cloud Computing Guidance for Data Protection Act complaince

ico.org.uk has compiled a good list of questions that any organization should consider before it decides to leverage cloud computing solutions. The article also gives some good examples to help the reader understand the consequences of sharing personal data on the cloud. It also serves as a good primer for Cloud Computing.

http://www.ico.org.uk/for_organisations/data_protection/topic_guides/online/~/media/documents/library/Data_Protection/Practical_application/cloud_computing_guidance_for_organisations.ashx

Ruminating on EDI

Despite being very old, EDI (Electronic Data Interchange) is still a dominant force in many industries; especially in healthcare. Organizations have invested so much in EDI, that it's not economically practical to shift to other standards such as XML.

A good primer on EDI is available at:
http://ondisplayapparel.com/whitepapers/What_is_EDI_Whitepaper-Taps.pdf

There are two popular standards for EDI: X12 and EDIFACT. X12 was designed as the standard for EDI transactions in US, and EDIFACT emerged out of X12 for international use. EDIFACT is a global EDI standard supporting multi-country and multi-industry exchange. There are a lot of differences between X12 and EDIFACT, one important difference being the fact that X12 assigns numeric values to documents whereas EDIFACT lists names or abbreviations.

In the EDI world, we have the concept of a VAN (Value Added Network). VANs are service providers that act as messaging hubs between trading partners and handling the EDI communication.

A EDI interchange message is made up of "Segments" and "Elements". Each segment begins with a two- or three-word identifier (ISA, GS, ST, N1, REF) and ends with a delimiter. The elements within each segment are also separated by a different delimiter. A good example for HIPAA X12 Claim transaction is available here. 

Good links to understand Health Insurance Domain

Was quite impressed with the simple and lucid language used on about.com to explain the basic of Health Insurance in US. The following articles are worth a perusal for anyone wanting to jump-start on the fundamental concepts in Healthcare.

HMOs vs. PPOs – What Are the Differences Between HMOs and PPOs?
What Is a Health Insurance Provider Network? and also this link.
What's the Difference Between Copay and Coinsurance? and also here.
Why Is Health Care So Expensive?
What Is Hospice Care?
What’s the Difference Between Medicare and Medicaid?
Why do we need healthcare reforms? and What are the dangers of having uninsured people?
What is ACA (Affordable Care Act) or ObamaCare? Another good link here.
What Is a Health Insurance Exchange?
Bronze, Silver, Gold & Platinum–Understanding the Metal-Tier System
What are Health Saving Account and Flexible Saving Accounts?
Understanding Claims Adjudication
What are ICD-9 or ICD-10 Codes?
What is HIPAA regulation around privacy of patient information?
Myths About HIPAA, Patients and Medical Records Privacy
Explanation of Benefits - Understanding Your EOB  : EOB goes to the member
What is ERA? : ERA goes to the provider during electronic payments. EDI 835
What is Individual Mandate?
What is MLR? (Medical Loss Ratio). Another link here.
What are Consumer Directed Health Plans (CDHP)?
What is the National Practitioner Data Bank (NPDB)?
What is SBC? (Summary of Benefits and Coverage)
What is provider credentialing?
What are tiered network health plans?
Who runs Medicare and Medicaid? Centers for Medicare & Medicaid Services (CMS)
What is OIC (Office of the Insurance Commissioner)?
What is medical necessity?
What does NCQA (National Committee for Quality Assurance ) do?
What Is the Coordination of Benefits?

Transactions in MongoDB

In my previous blog post, we had gone through some of the advantages of MongoDB in terms of schema flexibility and performance. The Metlife case study showed how we can quickly create a customer hub using MongoDB that supports flexible dynamic schemas.

Another added advantage of using MongoDB is that you don't have to worry about ORM tools, as there is no object-relational impedance mismatch. Also you don't have to worry about creating an application cache, as MongoDB be default uses all available memory for its working set.

But what about transactions? Any OLTP application would need full support for ACID to ensure reliability and consistency of data. The following articles shed good light on the transaction support in MongoDB.

http://docs.mongodb.org/manual/faq/fundamentals/
http://css.dzone.com/articles/how-acid-mongodb
http://blog.scrapinghub.com/2013/05/13/mongo-bad-for-scraped-data/

MongoDB only supports "Atomicity" at the document level. It's important to remember that we can have nested documents and MongoDB would support atomicity across the nested documents. But if we need multi-object transaction support, then MongoDB is not a good fit.

Also if your application needs to "join" objects frequently, then MongoDB is also not suitable in that respect. For e.g. loading reference data (static data) from master tables with the transaction data.  
MongoDB locks (Readers-Writer lock) are at the database level; i.e. entire database gets locked during a write operation. This can result in lock contention when you have a large number of write operations.

Looking at the pros-n-cons of MongoDB, IMHO it is best suited for heavy-read operations type of application. For e.g. a consolidated high performance read-only customer hub, a data store for content management systems, product catalogs in e-commerce systems, etc.

Log4javascript - Logging on the client side

Being an ardent fan of Log4J and Log4Net frameworks, I was pleased to see a port of the framework on Javascript for client side logging - log4javascript

With the plethora of pure JS web frameworks in the market today, it's very important to have a solid reliable logging framework on the client side (browser) that would enable us to debug problems easily. The default way to display log messages is in a separate popup window, featuring advanced search and filtering functionality, as well as a command line. You can also log to an in-page log4javascript console, to Firebug, to the browsers built-in error consoles or back to the server via Ajax POST.

I liked the OOTB Ajax appender that could be used to asynchronously post messages back to the server. We would need to write the server side code for the service. This is not included by default in the library.

A few organizations are using MongoDB to store log files. Having both client-side and server-side logs stored in a document database can be very useful for audit and debugging purposes. 

Ruminating on multicore CPUs and hyperthreading

My laptop has an i3 processor and I knew that i3 processors have 2 physical cores. But when I open Task Manager, I can see 4 CPUs. Even opening "device manager" shows 4 CPUs. So I was a bit confused on this.

A bit of research on the internet showed that the i3 processors have Hyper-Threading (HT) enabled by default. What that means is that the OS sees each 'physical' core as two 'logical cores'. This enables the operating system to schedule tasks to both the logical CPUs simultaneously.

The performance increases because whenever there is a cache miss or the CPU enters wait state due to a dependency, the other waiting thread can be put on the CPU core immendiately. This ensures optimal utilization of our CPU core resources.

So how is this different from traditional multithreading? In multi-threading the OS does the time-division multiplexing between multiple threads, whereas in HT the OS sees the core as two logical CPUs. Also in HT, when we talk about threads, it is hardware level threads. Each OS maps its OS level threads to the hardware threads.

NoSQL for Customer Hub MDM

The following article on informationweek is an interesting read on the use of MongoDB NoSQL for building a customer MDM solution.

http://www.informationweek.com/software/information-management/metlife-uses-nosql-for-customer-service/240154741

MongoDB being a document oriented NoSQL database has its core strength in maintaining flexible schemas and storing data as JSON or BSON objects. Lets look at the pros and cons of using MongoDB as a MDM solution.

1. One of the fundamental challenges faced is creating a customer hub is the aggregation of disparate data from a variety of different sources. For e.g. a customer could have bought a number of products from an insurance firm. Using a traditional RDBMS would entail complexities of joining the table records and fulfilling all the referential constraints of the data. Also each insurance product may have different fields and dimensions. Should we create a table for each product type? In MongoDB, you can store all the policies of the customer in one JSON object. You can store different types of policy for each customer with full flexibility and maintain a natural hierarchy (parent-child) of relationships. 


2. Another problem that Insurance firms face is that of legacy policy records. Certain insurance products such as Annuity have a long life period,but a lot of regulations and business needs change over the years and your old policy records may not have all the fields that are captured in new policy records. How do you handle such cases? Having a strict schema would not help and hence a solution like MongoDB offers the necessary flexibility to store spare data. 


3. MongoDB also has an edge in terms of low TCO for scalability and performance. Its auto-sharding capabilities enable massive horizontal scalability. It also supports OOTB memory-mapped files that is of tremendous help with the prominence of 64-bit computing and tons of available RAM. 

On the negative side, I am a bit concerned about the integrity of data in the above solution. Since there is no referential integrity, are we 100% confident on the accuracy of data? We would still need to use data profiling, data cleansing and data matching tools to find out unique customers and remove duplicates. 

Metlife is using this customer hub only for agents and has not exposed this data to the customers as there are concerns about data integrity and accuracy. But what if we need to enable the customer to self-service all his policies from a single window on the organizations portal ? We cannot show invalid data to the customer. 

Also from a skills perspective, MongoDB needs specialized resources.Its easy to use and develop, but for performance tuning and monitoring you need niche skills. 

Gender Diversity in the Architecture Field

Being passionate about gender diversity, I have always been concerned about the under-representation of women in the software architecture field. Over the years, I have endeavored to motivate and inspire my female colleagues to take up leadership roles in the technology stream; but in vain.

I have often introspected on the reasons why women don’t take or don’t make it to senior leadership roles in the enterprise architecture domain. Popular opinions range between the polarized extremes of “lack of interest” to “lack of competence” or both. I strongly beg to differ on the false assumption that women lack the logical skills to make good architects. In my career, I have seen brilliant women intellectuals with very strong programming and design skills. Women also tend to have better “EQ” (Emotional Intelligence) than men in general and this tremendously helps in areas such as decision-making, stakeholder communication and collaboration, conflict management, etc. So the “lack of competence” excuse is only for lame male chauvinists.

 I have mixed opinions on the “lack of interest” argument. Today we have compelling scientific evidence that proves that there are fundamental differences in the way the brains of men and women are hardwired. If you are not convinced on this, please peruse the books of John Gray (http://www.marsvenus.com). Many of his books were an eye-opener for me :). Considering these gender differences, in the way our brains are structured, can we make a generalized statement that most women are not passionate enough about cutting edge technology or software architecture? For e.g. when you get a new blue-ray player, or media server or any electronic gadget at home, who is the one to fiddle with it till all the functions are known? - the husband or wife? the son or daughter? Who watches family soap operas and who watches hi-tech action movies? Are men is general more interested in technology than women? Or is it because of lack of opportunities and gender bias? I don't have a clear answer, but I know for sure that mother nature has hardwired our brains differently. Family responsibilities and children upbringing is another challenge that must be forcing many women to make a choice on what's most important to them?

Maybe it’s time to change our preconceived notions about leadership and not equate it with aggressiveness and other ‘alpha-male’ characteristics? Lack of role models also proves to be detrimental in motivating women to pursue a technical career path in the architecture field. But this is a “chicken-n-egg” problem and an initial momentum is required to correct this.

Today’s world needs software architects with versatile skills and not just hard-core technical skills. We need architects who are better at brainstorming and collaboration, who can build on the ideas of others rather than aggressively push one’s own idea. In the Agile world, collaboration and communication is a key skill and women have a natural advantage in these areas.

What should be done to encourage more women to take up careers in software architecture and design field? What proactive steps can be taken to bridge this diversity gap? Your thoughts are welcome.