Friday October 26, 2007
Friday October 26, 2007 At the last day of my first innings at Sun - 6 years and 3 months after I joined right out of college.
From the word go, I got the opportunity to work with some really brilliant engineers on really cool ideas, technologies & products - and that was one thing which continued to be so throughout my career here.
I am joining Yahoo! research engineering in Bangalore on monday ...
I can be reached at
mail : mridulm80 <at> yahoo.com
xmpp: mridul <at> gmail.com
ymsgr: mridulm80
aim: mridulm80
... thats all folks and thanks to all !
Wednesday August 08, 2007 We recently submitted a proposal to XSF to standardize what was a proprietary implementation for message moderation ... which we have had for a couple of years now. The other effort which we should probably kick off soon is file transfer to multi-user chat room - this is another custom implementation which we have been carrying on for a few releases now ...
This has come up time and again in JIG as part of other threads, and yet we never got much farther that 'yep, we should document it'. I know of atleast one other server who have their own implementation to support this .... so maybe it is time we standardized this !
To give a basic idea of what we do:
Currently, our client supports IBB - so does the server, and the muc (which is part of the server). So idea is simple - essentially stream the data from the client to the server, store it locally (if content filtering, etc is enabled) post-process it (if enabled), and then the component streams it to the recipients.
As all the participants who can understand this are Sun IM clients - we just use IBB for now.
The caveats are obvious - potentially very heavy server traffic due to the IBB. But the main advantages are:
File transfer in general was unfortunately not a very strong point in xmpp, and we are still working on the details ... so I am not very sure if we have to wait for the churn to subside, before proceeding - or start work on this in parallel, so that the requirements for multicasting file(s) to a room are kept in mind while tackling the file transfer problem.
This has come up time and again in JIG as part of other threads, and yet we never got much farther that 'yep, we should document it'. I know of atleast one other server who have their own implementation to support this .... so maybe it is time we standardized this !
To give a basic idea of what we do:
Currently, our client supports IBB - so does the server, and the muc (which is part of the server). So idea is simple - essentially stream the data from the client to the server, store it locally (if content filtering, etc is enabled) post-process it (if enabled), and then the component streams it to the recipients.
As all the participants who can understand this are Sun IM clients - we just use IBB for now.
The caveats are obvious - potentially very heavy server traffic due to the IBB. But the main advantages are:
- The sender transmits only a single copy of the file - in comparison to a p2p model (imagine streaming a 512 KB file to a muc with 100 participants from a moderately low b/w connection).
- It will always work for all cases - irrespective of firewall, nat, etc. IBB is the lowest common denominator.
- Ability to analyze the content before sending it on to the recipients - including the possibility of archiving it (for participants who might join the room 'later').
File transfer in general was unfortunately not a very strong point in xmpp, and we are still working on the details ... so I am not very sure if we have to wait for the churn to subside, before proceeding - or start work on this in parallel, so that the requirements for multicasting file(s) to a room are kept in mind while tackling the file transfer problem.
Wednesday August 01, 2007 
Saturday July 28, 2007 We recently submitted two proposals for message moderation - Message Moderated Conferences and Managing message moderators in MUC rooms - both of which are extensions to Multi-User Chat (muc) spec.
Taken together, they allow a message moderation system to be put in place. It has been a conscious decision not to introduce another set of acl's for this, but to reuse the affiliations and roles specified in muc. I will try to go through the basic intent and design decisions behind the specs.
If message moderation is enabled, these occupants would be able to request the room for approving the messages they want to post - and on approval, room would publish them to all occupants of the room. Simple scenarios would be a celebrity chat, moderated webcast/presentation, etc.
The spec does not deal with how the message moderation happens - just the interaction between a occupant and the room - how they interact, state changes, notifications, etc. So in effect, you could have any number of backend moderation implementations - but the interface between the occupant and the room will remain the same.
The basic flow is as follows :
The actual moderation might be quite involved, with multiple moderators/moderation modules chained, clustered/distributed rooms, etc in complex deployments - but the occupant will have a simple and clean interface to communicate with the room (room's bare jid actually).
The way we designed it, hopefully this will someday get rolled into XEP 45 itself
It defines how a room moderator/owner can become a message moderator (and other state changes), how they notify room of the moderation decision, and how the room is expected to manage these moderators and submitted messages for moderation.
Both of these specs are based on our implementation - though what we support currently is quite different from what we have submitted.
As a sidenote : It should be noted that, in both the specs we have tried to make sure that the interface between the various entities in question remains as simple as possible - while not precluding more complicated scenarios .. like usecases which are not specifically related to users chatting over a moderated conference (for example an approval workflow) - there could be custom configurations with extensions to room configurations which allow for more complicated scenarios while keeping the actual client interfaces (at both submitter and moderator side) constant.
Taken together, they allow a message moderation system to be put in place. It has been a conscious decision not to introduce another set of acl's for this, but to reuse the affiliations and roles specified in muc. I will try to go through the basic intent and design decisions behind the specs.
Message Moderated Conferences
This proposal specifies how a conference which has message moderation enabled would interact with participants - particularly, participants who dont have 'voice' or the ability to post message to the room.If message moderation is enabled, these occupants would be able to request the room for approving the messages they want to post - and on approval, room would publish them to all occupants of the room. Simple scenarios would be a celebrity chat, moderated webcast/presentation, etc.
The spec does not deal with how the message moderation happens - just the interaction between a occupant and the room - how they interact, state changes, notifications, etc. So in effect, you could have any number of backend moderation implementations - but the interface between the occupant and the room will remain the same.
The basic flow is as follows :
- Occupant sends message to room for approval.
- Room assigns a message id to the submitted message (which is then used to identify the submitted message for all further interaction, in this and other moderation related specs) and returns that to the user with message in pending state.
- After the backend moderation system decides on the message, room will inform the submitted about the 'decision' - approved, rejected (or error).
- If approved, room will then multicast it to all the occupants.
The actual moderation might be quite involved, with multiple moderators/moderation modules chained, clustered/distributed rooms, etc in complex deployments - but the occupant will have a simple and clean interface to communicate with the room (room's bare jid actually).
The way we designed it, hopefully this will someday get rolled into XEP 45 itself
Managing message moderators in MUC rooms
This spec defines one possible way in which message moderation can actually be implemented - where the moderation is done by participants who have sufficient rights (owner's and moderators of the room).It defines how a room moderator/owner can become a message moderator (and other state changes), how they notify room of the moderation decision, and how the room is expected to manage these moderators and submitted messages for moderation.
Both of these specs are based on our implementation - though what we support currently is quite different from what we have submitted.
As a sidenote : It should be noted that, in both the specs we have tried to make sure that the interface between the various entities in question remains as simple as possible - while not precluding more complicated scenarios .. like usecases which are not specifically related to users chatting over a moderated conference (for example an approval workflow) - there could be custom configurations with extensions to room configurations which allow for more complicated scenarios while keeping the actual client interfaces (at both submitter and moderator side) constant.
Tuesday March 27, 2007 some thoughts on xmpp s2s availability and failover : and lack of protocol support for it.
[Read More]
Sunday December 17, 2006 Discusses some rationale behind thoughts on why strict in order delivery maynot be a 'good thing'.
[Read More]
Tuesday November 28, 2006 One of the things which we introduced with the previous interim release
was the concept of server pool.The admin can have a combination of
disparate set of boxes - solaris sparc/x86, linux and logically combine
them to create a single XMPP deployment.Ofcourse, one deployment could
support multiple hosted domains - so typically it is a single
deployment : not really a single domain which is hosted on a pool.
For the purpose of minimising internode communication, we have also introduced to concept of an xmpp aware load balancer called redirect server.The actual redirection mechanism is an spi and can be customized, but there are a bunch of out of the box methods - including ones which allow for roster based logical grouping of contacts to nodes.The server pool will redistribute any new load across itself in the face of failures ... So, introducing redundency into the pool not only helps in performance of a single node under normal operation, but allows the pool to operate smoothly when you are faced with network, power, etc outages.
Taken together this results in some interesting usecases.
There is minimal network overhead and yet high degree of failover and scalability can be achieved - we have not really tested to reach the limits of how 'large' a server pool can be .... This essentially means that a service provider can not only support a large number of hosted domains, but also provide high availability for all of them with a single deployment.
As far as the end user is concerned, there is no difference whether he is talking to a server pool or to a single server: the behaviour is uniform and spec compliant.
If you consider the distributed nature of features like pubsub, muc, etc in the server pool - it essentially means that you do not have the concept of failure: unless every node in the pool goes down that is
So essentially, you can use the server as a near-realtime messaging middleware with very high availability and scalability.
And yes, the next release we have does include expirimental support for caps and pep at the server and in client api ... might not be the latest version of the spec though. But developers wishing to hack at it are welcome : this impl is not very rigourously tested - so I am not going to popularise it, but we have used it for avataar and seen it working beautifully
!
The api which is hosted at collab project of netbeans has the client side support for this (:pserver:anoncvs-AT-cvs.netbeans-DOT-org:/cvs , collab/service/src/org/netbeans)
Btw, did I mention that we have always supported privacy lists ?
For the purpose of minimising internode communication, we have also introduced to concept of an xmpp aware load balancer called redirect server.The actual redirection mechanism is an spi and can be customized, but there are a bunch of out of the box methods - including ones which allow for roster based logical grouping of contacts to nodes.The server pool will redistribute any new load across itself in the face of failures ... So, introducing redundency into the pool not only helps in performance of a single node under normal operation, but allows the pool to operate smoothly when you are faced with network, power, etc outages.
Taken together this results in some interesting usecases.
There is minimal network overhead and yet high degree of failover and scalability can be achieved - we have not really tested to reach the limits of how 'large' a server pool can be .... This essentially means that a service provider can not only support a large number of hosted domains, but also provide high availability for all of them with a single deployment.
As far as the end user is concerned, there is no difference whether he is talking to a server pool or to a single server: the behaviour is uniform and spec compliant.
If you consider the distributed nature of features like pubsub, muc, etc in the server pool - it essentially means that you do not have the concept of failure: unless every node in the pool goes down that is
So essentially, you can use the server as a near-realtime messaging middleware with very high availability and scalability.
And yes, the next release we have does include expirimental support for caps and pep at the server and in client api ... might not be the latest version of the spec though. But developers wishing to hack at it are welcome : this impl is not very rigourously tested - so I am not going to popularise it
, but we have used it for avataar and seen it working beautifully
!The api which is hosted at collab project of netbeans has the client side support for this (:pserver:anoncvs-AT-cvs.netbeans-DOT-org:/cvs , collab/service/src/org/netbeans)
Btw, did I mention that we have always supported privacy lists ?
Saturday November 18, 2006 File transfer to a Multi user chat is not defined in the current set of xmpp extensions.
Though this does not really describe what our server/client does today, this post discusses a simple means to achieve this required functionality.
[Read More]
Wednesday November 15, 2006 Some random thoughts on presence probes in rfc 3921 and the xmpp bis specs ... just random rambling, I could be grossly wrong !
[Read More]
Monday November 13, 2006 A small summary of primary features to expect in security and authentication in the upcoming release of Sun's IM server
[Read More]
Friday November 10, 2006 Elaborated the idea posted in the previous post along with some clarifications and corrections.
[Read More]
Wednesday November 08, 2006 Discusses a potential proposal we are considering to implement which will minimise the interserver traffic tremendously without compromising on security, responsibility and accountability.
[Read More]
Tuesday September 12, 2006 Yep , been ages since I did anything at xmpp-im-client : and all blame to be laid on me.
One thing let to another and I was neck deep in fixing the server up quite a bit !
The next release when it does come out is going to be quite interesting
Some very interesting usecases are going to be possible ... patience for a couple of months more !
So , I will need to tie up loose ends for a couple of more weeks : and then hopefully I will be able to get enough of my life back to actually start working on the project.
One thing let to another and I was neck deep in fixing the server up quite a bit !
The next release when it does come out is going to be quite interesting
Some very interesting usecases are going to be possible ... patience for a couple of months more !
So , I will need to tie up loose ends for a couple of more weeks : and then hopefully I will be able to get enough of my life back to actually start working on the project.
Monday September 04, 2006 A very clear and concise post by Neal Gafter on the use , differences and benifits of closures here.
When I read the initial proposal , it was slightly confusing.
I felt that the return syntax smelt like goto's , ability to access non-final local variables was worrisome , etc.
The ideas on the last two posts (here and here) are very instructive and clarify things quite a bit.
The synchronous usecase is something I have tried time and again to solve within my code - usually inefficiently with hacked up callbacks : closures would be a much more cleaner fix for this problem.
When I read the initial proposal , it was slightly confusing.
I felt that the return syntax smelt like goto's , ability to access non-final local variables was worrisome , etc.
The ideas on the last two posts (here and here) are very instructive and clarify things quite a bit.
The synchronous usecase is something I have tried time and again to solve within my code - usually inefficiently with hacked up callbacks : closures would be a much more cleaner fix for this problem.
Monday August 28, 2006 As more and more websites and applications start moving to using the AJAX paradigm , people are going to hit this problem of how to use HTTP as a transport 'reliably'.
What I mean is , HTTP is essentially similar to a one way datagram approach to communication.
You send request A , wait for response A - move on.
Since this need not be a direct connection from client to server (even if it was) , the request could fail to be delivered , response could be 'lost' , connection errors , etc could intervene.
If they are idempotent methods being made , then you can reissue them in case the request fails for whatever reason.
If not , things are slightly trickier - but there is some sort of support in most frameworks on how to handle this currently.
Add HTTP/1.1 persistent connections to this mix and things get really interesting - one socket fails : and a whole bunch of requests are lost.
Also, AJAX typically issues POST in most cases - a non-idempotent method.
Add to it the requirement to 'poll' the backend for data or updates and you have a interesting problem : you are trying to simulate a HTTP client into being something similar to a bidirectional stream : simulate is a key word here , mind you !
So you have requirements to 'manage HTTP as a transport' coming both at the client side and at the server side.
In the XMPP world , when support for HTTP was initially added the first approach was a pure polling solution.
As can be guessed , they later developed the HTTP Binding solution under JEP 124.
This is a very very interesting extension and I believe is the key to solving most of the problems above.
Though defined for XMPP , it is a solution to essentially act as a transport for any xml based protocol : only requirement, one or more "full xml stanza's" are exchanged - not partial xml data.
It solves the problems of retransmission , session management (no need for cookie's , url rewriting , etc) , support against replay attacks , etc.
Heck , we have XMPP - a purely xml based streaming protocol working on top of this successfully !
How does this help AJAX ?
All the browser based XMPP clients which support this JEP have some variants or other javascript library in place to actually do all this.
So , potentially you could have :
<your application> --> <jep 124 client side js library> --- HTTP ---> <HTTP Bind gateway> <-- Custom protocol --> <Your server>
With suitable modification (IF your app is not XMPP that is ... why not move to it? Read prev post on some advantages) , you can use the ideas from JEP 124 to solve a whole bunch of problems with using HTTP as a reliable transport.
Add to it the fact that you already have server side and client side opensource component's available to support this - and you have the perfect solution just waiting to be used.
Happy coding !
What I mean is , HTTP is essentially similar to a one way datagram approach to communication.
You send request A , wait for response A - move on.
Since this need not be a direct connection from client to server (even if it was) , the request could fail to be delivered , response could be 'lost' , connection errors , etc could intervene.
If they are idempotent methods being made , then you can reissue them in case the request fails for whatever reason.
If not , things are slightly trickier - but there is some sort of support in most frameworks on how to handle this currently.
Add HTTP/1.1 persistent connections to this mix and things get really interesting - one socket fails : and a whole bunch of requests are lost.
Also, AJAX typically issues POST in most cases - a non-idempotent method.
Add to it the requirement to 'poll' the backend for data or updates and you have a interesting problem : you are trying to simulate a HTTP client into being something similar to a bidirectional stream : simulate is a key word here , mind you !
So you have requirements to 'manage HTTP as a transport' coming both at the client side and at the server side.
In the XMPP world , when support for HTTP was initially added the first approach was a pure polling solution.
As can be guessed , they later developed the HTTP Binding solution under JEP 124.
This is a very very interesting extension and I believe is the key to solving most of the problems above.
Though defined for XMPP , it is a solution to essentially act as a transport for any xml based protocol : only requirement, one or more "full xml stanza's" are exchanged - not partial xml data.
It solves the problems of retransmission , session management (no need for cookie's , url rewriting , etc) , support against replay attacks , etc.
Heck , we have XMPP - a purely xml based streaming protocol working on top of this successfully !
How does this help AJAX ?
All the browser based XMPP clients which support this JEP have some variants or other javascript library in place to actually do all this.
So , potentially you could have :
<your application> --> <jep 124 client side js library> --- HTTP ---> <HTTP Bind gateway> <-- Custom protocol --> <Your server>
With suitable modification (IF your app is not XMPP that is
... why not move to it? Read prev post on some advantages) , you can use the ideas from JEP 124 to solve a whole bunch of problems with using HTTP as a reliable transport.Add to it the fact that you already have server side and client side opensource component's available to support this - and you have the perfect solution just waiting to be used.
Happy coding !