Mytec
Grizzly 2.0: Streaming and Messaging
Originally, when designing Grizzly 2.0 Connection API, we were thinking that Streams could cover all the possible scenarios developers may have. and could be easily used with TCP and UDP transports. But when started to implement UDP transport, we came to conclusion, that it's not actually true. For sure UDP is message oriented protocol, but thought we assumed it will be easy to emulate that using Streams like:
streamWriter.writeInt(...);
streamWriter.writeLong(...);
streamWriter.flush();
for connected UDP Connection, or
streamWriter.writeInt(...);
streamWriter.writeLong(...);
streamWriter.flush(dstAddress);
for non-connected UDP Connections. Similar trick could be done for reading UDP messages.
So, from API point of view, Streams could work fine even for message-oriented protocols, but the problem appears, when we build UDP server (have non-connected UDP socket). In this case server may receive packets from different clients, but as Streams are not thread-safe, we can process only one single client packet at the time and block other clients, until first packet will be completely processed. This fact creates sensitive performance issue for UDP server, because many UDP packets may get lost because of big delays in processing. For such a usecase we need possibility to not block the connection, when processing incoming message, but make the connection available to process next message. This is not doable with Streams, because of mentioned thread-safety limitation, so in Grizzly 2.0 M2 we've added message-based API for Connections.
Message-based API:
Message-based API is reachable via Connection interface, which is now extending Readable and Writable interfaces, and is represented by set of read and write methods:
- Future<ReadResult<Buffer, L>> read(...);
- Future<WriteResult<Buffer, L>> write(...);
where <L> represents source/destination address type (SocketAddress for NIO TCP and UDP transports).
Ok, with Connections it should be straightforward, how we can use Message-based API, what about FilterChains? In the original Grizzly samples, TransportFilter, which we add first to a FilterChain, provided us StreamReader/StreamWriter objects to be used by next Filters in chain. So TransportFilter was oriented to work just in Stream mode. Now it's possible to create TransportFilter, which will work in Message mode:
transport.getFilterChain().add(new TransportFilter(TransportFilter.Mode.Message));
if we add TransportFilter like above, then it will not provide Streams, which could be used by the rest of FilterChain, but message (Buffer), which could be accessed via FilterChainContext:
Buffer message = (Buffer) context.getMessage();
so next Filters in chain should deal directly with message, not Streams.
Here is the simple example of using Message-based API for UDP echo server.
Stream-based API:
We discussed Stream-based API in one of my previous blogs. So here I just want to add, that for TCP transport we recommend to use Stream API, because it makes implementation easier for most of usecases and takes care about internal buffer management, optimizing memory allocation etc.
Let's try to make some summary for Streaming and Messaging API. For sure in each particular case, it's up to you to decide which API to use. Here are just our recommendations:
- Streams API is recommended to be used with TCP transport and could be also used for connected UDP Connections;
- Message API is recommended to be used with non-connected UDP Connections (UDP servers);
Posted at 02:44PM May 27, 2009 by oleksiys in Grizzly | Comments[0]
Grizzly 2.0 M2 Release
Next Grizzly 2.0 milestone is reached.
Here is the list of new features available with Grizzly 2.0 M2:
- UDP transport,
- New Connection API, which lets Connections to operate in "message" mode. This API fits better to message oriented transports like UDP,
- Performance improvements,
- Codec API and specifically SSL and FilterChain Codecs,
- Extended Transport API to support multi-binding, added support for unbind(),
- Significantly improved documentation,
- All-in-one OSGi bundle for framework, http and http-servlet available. You can launch them by just doing java -jar grizzly-<module>
Posted at 04:51PM May 26, 2009 by oleksiys in Grizzly | Comments[1]
Grizzly 2.0 StreamReader/StreamWriter API
When you work with Grizzly 2.0, you'll most probably come to situation, when you need to read or write some data on network :))
Core Grizzly 2.0 I/O API is based on 3 main entities:
- Connection, which represents any kind of Transport connection. As example for TCP NIO Transport, one Connection represents one NIO SocketChannel.
- StreamReader represents Connection's input stream, using which it's possible to read data from Connection.
- StreamWriter represents Connection's output stream to write data to Connection.
So to read data from Connection, we need to read data from Connection's StreamReader: connection.getStreamReader().readXXX(); The similar step, but with StreamWriter, we need to make in order to write some data to Connection: connection.getStreamWriter().writeXXX(...);
StreamReader and StreamWriter API has a set of readXXX(), writeXXX() methods to work with Java primitives and arrays of primitives. For example to write float value to Connection we call corresponding StreamWriter write method: connection.getStreamWriter().writeFloat(0.3f);
How Streams could work in non-blocking?
In Grizzly 2.0 we can work with StreamReader and StreamWriter either in blocking or non-blocking mode. The mode could be checked and set using following methods: isBlocking(), setBlocking(boolean).
When Streams operate in blocking mode - all their methods, which may work asynchronously (return "Future") will work in blocking mode, and returned Future will always have ready result.
As for non-blocking mode, ok let's start from...
StreamReader.
In order to use StreamReader in non-blocking mode - we may want to check if StreamReader has enough data to be read: streamReader.availableDataSize(); So once StreamReader has enough data - we can safely read it without blocking.
It is also possible to ask StreamReader to notify us once it will have enough data, or provide any other custom Condition, which StreamReader should check each time new data come - and notify us, once this Condition is met. For example:
Future<Integer> future = streamReader.notifyAvailable(10, completionHandler);
StreamReader returns Future object, which will mark as done, once StreamReader will have available 10 bytes for reading. At the same time we pass the CompletionHandler, which will be notified, once StreamReader will have 10 bytes available. So it's possible to have poll and push-like notifications with StreamReader.
We can ask StreamReader to get notification, when more complex conditions are met.
Future<Integer> future = streamReader.notifyCondition(customCondition, completionHandler);
This way we implemented SSL handshake mechanism, so SSLStreamReader notifies us, when handshake status becomes NEED_WRAP.
StreamWriter.
Non-blocking mode for StreamWriter means, that stream flushing will be done non-blocking. For example:
Future<Integer> future = streamWriter.flush();
where flush() operation returns Future<Integer>, which could be used to check if bytes were flushed and how may bytes were written on Connection. It is also possible to pass CompletionHandler to flush() operation, which will be notified by StreamWriter, once bytes will be flushed.
Posted at 02:34PM Apr 22, 2009 by oleksiys in Grizzly | Comments[4]
Grizzly 2.0 M1 Release
Finally we're ready to make first milestone release of Grizzly 2.0
The M1 release has 4 modules:
- Framework: Grizzly 2.0 core
- RCM: Resource Consumption Filter implementation, based on Grizzly 2.0 API
- HTTP: Lightweight Web container, based on Grizzly 2.0 API
- Servlet: Lightweight Servlet container, based on Grizzly 2.0 API.
There are 2 modules, which provide examples of basic features, provided by Grizzly 2.0:
- Framework-samples: "echo" client/server application, SSL example, Connection lifecycle control using Grizzly 2.0 API, custom I/O Strategy example.
- HTTP-samples: simple Grizzly lightweight webserver using Grizzly core API (WebFilter) and higher level GrizzlyWebServer API. Both HTTP and HTTPS scenarios.
Following new features are included in this release:
- New I/O API, based on Grizzly StreamReader and StreamWriter.
- Extensible memory management API, which makes possible to implement either simple or complex memory management logic and use it with Grizzly.
- Extensible Strategy API for processing Connection events. Grizzly bundles 4 built-in strategies:
- SameThreadStrategy, which processes all I/O events in the same thread they were got (selector thread), so no additional worker threads are used. This strategy could be very useful and optimal for processing small amount of clients, and lets server have great response time
- LeaderFollowerStrategy. Performs I/O processing in the same thread (selector thread), and delegates selector polling logic to another thread.
- WorkerThreadStrategy. Performs all the I/O events in separate worker thread.
- SimpleDynamicStrategy. Dynamically applies one of the above 3 strategies, depending on number of SelectionKeys selected last time.
- WebFilter. Grizzly lightweight webcontainer implemented as Filter.
- GrizzlyWebServer. High level API to work with Grizzly lightweight webcontainer.
Later I will write separate blogs describing each feature....
Stay tuned...
Posted at 10:49PM Apr 21, 2009 by oleksiys in Grizzly | Comments[1]
Glassfish V3: Asynchronous HTTP responses
We've completed integration of the latest Grizzly 1.9.0 binaries to Glassfish v3.
One of the most interesting features, which now become available in GFv3, is asynchronous HTTP responses. Recently I described, how this feature could be used with the standalone Grizzly 1.9.0. Now it is also applicable for GFv3.
To enable asynchronous HTTP responses in GFv3, we need to add following line to the domain.xml (/domain/configs/config/java-config):
<jvm-options>-Dcom.sun.grizzly.http.asyncwrite.enabled=true</jvm-options>
It's also possible to change the size of ByteBuffer pool, used for cloning ByteBuffers before asynchronous write (details). Again we can do this by adding domain.xml (/domain/configs/config/java-config) property:
<jvm-options>-Dcom.sun.grizzly.http.asyncwrite.maxBufferPoolSize=NEW_SIZE</jvm-options>
PS: This feature has been added after GFv3 Prelude release, so is currently available on GFv3 trunk only.
Posted at 11:29AM Dec 09, 2008 by oleksiys in Grizzly | Comments[0]
Simple Grizzly 2.0
From time to time I see mails from people, who are looking forward to use Grizzly 2.0, but don't know where to start from. Currently we don't have much documentation on that and mainly I have to point people to unit tests code to give them some idea about Grizzly 2.0
Here I'll try to show the very simple scenario, how Grizzly 2.0 could be used. I will not show FilterChains, CallbackHandlers... nothing. I will show how Grizzly 2.0 could be used instead of plain Sockets. So, even if you have just basic understanding how Java Sockets work - you'll understand how Grizzly 2.0 could be used instead.
Why use Grizzly 2.0 instead of Sockets?
- Grizzly 2.0 provides very simple API, based on Future, CompletionHandler API, which lets you work asynchronously (in non-blocking manner) with connections.
- Grizzly 2.0 uses asynchronous read and write queues, which means you don't have to care about simultaneous reads and writes from different threads.
- Understand better Grizzly 2.0 API, which will help you to write more complex Grizzly based systems.
Client, based on Grizzly 2.0:
TCPNIOTransport transport = TransportFactory.getInstance().createTCPTransport();
// Enable standalone mode
transport.setProcessorSelector(new NullProcessorSelector());
// Start transport
transport.start();
// Connect to a server
ConnectFuture connectFuture = transport.connect(host, port);
Connection connection = connectFuture.get();
// Initialize data, which will be sent
MemoryManager memoryManager = transport.getMemeoryManager();
Buffer buffer = MemoryUtils.wrap(memoryManager, "Hello World!", Charset.forName("UTF-8"));
// Send data (async write queue will be used underneath)
Future writeFuture = connection.write(buffer);
// We can wait until write will be completed
writeFuture.get(timeout, TimeUnit.SECONDS);
// Initialize receive buffer
Buffer recieveBuffer = memoryManager.allocate(SIZE);
//Read response
Future readFuture = connection.read(receiveBuffer);
/*
* Tip. If we need to read whole buffer (readFully), use following:
* Future readFuture = connection.read(receiveBuffer, null, null, MinBufferSizeCondition(receiveBuffer.remaining());
*/
readFuture.get(timeout, TimeUnit.SECONDS);
......................
connection.close();
Socket-like server, based on Grizzly:
TCPNIOTransport transport = TransportFactory.getInstance().createTCPTransport();
// Enable standalone mode
transport.setProcessorSelector(new NullProcessorSelector());
// Bind transport to listen on specific port
transport.bind(PORT);
// Start transport
transport.start();
while(!transport.isStopped()) {
// Accept new client connection
AcceptFuture acceptFuture = transport.accept();
Connection connection = acceptFuture.get();
// Process accepted connection (may be in separate thread)
processAcceptedConnection(connection);
}
From the samples above we can see, that Grizzly 2.0 provides API similar to Java Socket, but it lets us work with connections asynchronously! May be Grizzly need more initialization lines than Sockets:
- TCPNIOTransport transport = TransportFactory.getInstance().createTCPTransport();
- transport.setProcessorSelector(new NullProcessorSelector();
The "1" just initializes the TCP transport.
The "2" configures transports, so it will be used in standalone mode. FilterChains, CallbackHandlers and other possible Processors will not be invoked.
And again, if you want working example, how Grizzly could be used in standalone mode (Socket-like manner), please take a look here.
Posted at 06:15PM Dec 04, 2008 by oleksiys in Grizzly | Comments[2]
Grizzly 1.9: ExecutorService as common thread pool interface
Yesterday we've released Grizzly 1.9
One of the biggest releases we had so far. Here is the announce from Jean-Francois with the complete list, what was added to the new release.
Here I'll provide more details on new Grizzly1.9 thread pool API (actually there is nothing new for those, who uses java.util.concurrent objects). Since version 1.9 we stop to support Pipeline API and move to more standard and known ExecutorService.
So, now developer should not write thread pool wrapper, based on Pipeline, in order to use custom Thread pool implementation, but directly use own ExecutorService implementation, which will be responsible for worker threads lifecycle.
ExecutorService, as thread pool API, simplifies Grizzly integration process with existing platforms/frameworks, which may have own thread pools (based on ExecutorServices) and now are able to share the same thread pool with Grizzly.
Here is example how Grizzly thread pool could be configured and set:
Controller controller = new Controller();
ThreadPoolExecutor threadPool = new ThreadPoolExecutor(corePoolSize, maximumPoolSize,
keepAliveTime, unit, workQueue, new WorkerThreadFactory());
threadPool.setMaximumPoolSize(5);
controller.setThreadPool(threadPool);
........................................
private class DefaultWorkerThreadFactory implements ThreadFactory {
public Thread newThread(Runnable r) {
Thread thread = new WorkerThreadImpl(null, "WorkerThread", r, initialByteBufferSize);
thread.setPriority(priority);
return thread;
}
}
Posted at 04:43PM Dec 04, 2008 by oleksiys in Grizzly | Comments[0]
Grizzly 1.9.0: Asynchronous HTTP responses
In Grizzly 1.9.0, which will be released very soon, we've implemented new feature for HTTP module.
Now it is possible to send HTTP responses asynchronously without blocking a worker thread. What does it mean and which advantages we get?
In earlier Grizzly versions, when we sent HTTP response, a current thread was blocked until whole the response will be written on network. This is fine, when response is relatively small and server is not overloaded with processing HTTP requests.
But in case, when server is under load and is not able to write HTTP response fast... We block on thread and wait, when OS will become ready to write next chunk of data on network, so write operation becomes a bottleneck for our server scalability.
In Grizzly 1.9 it is possible to leverage the advantages, proposed by Grizzly asynchronous write queue. So now, if channel can not write more data on wire, instead of blocking on a thread, we add the HTTP response message to a write queue. Asynchronous write queue will be processed, once OS will signal, that channel is available for writing.
The asynchronous mode for HTTP responses is turned off by default. So here are ways, how it could be turned on:
1) Programmatically, using SelectorThread
SelectorThread selectorThread = new SelectorThread();
selectorThread.setPort(PORT);
selectorThread.setAsyncHttpWriteEnabled(true);
2) Using system property
-Dcom.sun.grizzly.http.asyncwrite.enabled=true
Though asynchronous mode for HTTP responses is very useful and has huge advantages, comparing to blocking mode, it has one "detail" :), which we have to be careful with. I'm talking about ByteBuffer cloning.
At time, when we can not write more data on channel, we add the ByteBuffer to asynchronous write queue, which means we can not continue to work with this ByteBuffer, until it will be released from asynchronous write queue. So, to process next HTTP request, we have to create new ByteBuffer. So, basically to increase server scalability by using asynchronous write queue, we pay memory. In Grizzly 1.9 we use simple ByteBuffer pool with limited size, to avoid creating new ByteBuffers all the time. The size of the ByteBuffer pool could be tuned.
1) Programmatically:
SocketChannelOutputBuffer.setMaxBufferPoolSize(int size);
2) Using system properties:
-Dcom.sun.grizzly.http.asyncwrite.maxBufferPoolSize=<size>
Posted at 06:23PM Dec 02, 2008 by oleksiys in Grizzly | Comments[4]
Grizzly 2.0: SSL support
Recently we've added SSL support for Grizzly 2.0
Unlike Grizzly 1.x, Grizzly 2.0 doesn't have special transport, called SSL or TLS. The SSL support is implemented using new Transformer API we've introduced, which could be used either standalone or within FilterChain.
Here is brief description of classes:
SSLEncoderTransformer: encodes plaintext input Buffer into TLS/SSL encoded output Buffer.
SSLDecoderTransformer: decodes TLS/SSL encoded Buffer into plaintext data Buffer.
SSLCodec: incapsulates encoder and decoder transformers together with SSL configuration.
As I mentioned, it is possible to use SSL both in standalone and within FilterChain.
1) Standalone
In standalone mode, developer should implicitly initialize SSL connection by executing SSL handshake. Then it's possible to use Connection I/O methods: read/write to send or receive data.
Connection connection = null;// Initiate the SSLCodec
SSLCodec sslCodec = new SSLCodec(createSSLContext());
TCPNIOTransport transport = TransportFactory.instance().createTCPTransport();
try {
transport.bind(PORT);
transport.start();
// Connect client
ConnectFuture future = transport.connect("localhost", PORT);
connection = (TCPNIOConnection) future.get(10, TimeUnit.SECONDS);
// Run handshake
Future handshakeFuture = sslCodec.handshake(connection);
// Wait until handshake will be completed
handshakeFuture.get(10, TimeUnit.SECONDS);
MemoryManager memoryManager = transport.getMemoryManager();
Buffer message = MemoryUtils.wrap(memoryManager, "Hello world!");
// Write the message with SSLCodec.getEncoder() parameter.
Future writeFuture = connection.write(message, sslCodec.getEncoder());
writeFuture.get();
// Obtain the Buffer, which corresponds to the SSLEngine requirements.
Buffer receiverBuffer = SSLResourcesAccessor.getInstance().obtainAppBuffer(connection);
// Read the message with SSLCodec.getDecoder() parameter
Future readFuture = connection.read(receiverBuffer, sslCodec.getDecoder());
.....................................................
2. FilterChain mode
In FilterChain mode, developer should just add SSLFilter to the FilterChain. The SSLFilter itself has SSLCodec, which in its turn has SSL encode/decode transformers and SSL configuration.
Connection connection = null;SSLCodec sslCodec = new SSLCodec(createSSLContext());
TCPNIOTransport transport =
TransportManager.instance().createTCPTransport();
transport.getFilterChain().add(new TransportFilter());
// Add SSLFilter
transport.getFilterChain().add(new SSLFilter(sslCodec));
transport.getFilterChain().add(new EchoFilter());
try {
transport.bind(PORT);
transport.start();
...................
The last thing, I wanted to mention, is SSL configuration. How we can configure SSL?
As I told SSLCodec represents a core of SSL processing, it contains encoder/decoder Transformers and SSL configuration. In order to configure SSLCodec, it is possible to pass ready SSLContext, which could be created in your custom code, or use Grizzly 2.0 utility class SSLContextConfigurator.
Here is example, how SSLContextConfigurator could be used:
SSLContextConfigurator sslContextConfigurator = new SSLContextConfigurator();
URL cacertsUrl = cl.getResource("ssltest-cacerts.jks");
if (cacertsUrl != null) {
sslContextConfigurator.setTrustStoreFile(cacertsUrl.getFile());
}
URL keystoreUrl = cl.getResource("ssltest-keystore.jks");
if (keystoreUrl != null) {
sslContextConfigurator.setKeyStoreFile(keystoreUrl.getFile());
}
return sslContextConfigurator.createSSLContext();
If you have any question - pls. ask them on grizzly mailing lists :)
Posted at 04:49PM Dec 02, 2008 by oleksiys in Grizzly | Comments[0]
Recent Grizzly 2.0 updates
Recently we made several updates in Grizzly 2.0:
1) introduced Buffer interface, which will be common for all kind of buffers, used in Grizzly
2) changed read/write API according to what proposed Jeanfrancois. Instead of read/write, readAsync/writeAsync, readNow/writeNow methods - now we have just read/write. The I/O mode (blocking/async) could be changed, using Transport.configureBlocking(boolean) for whole transport, or Connection.configureBlocking() for specific connection.
3) completed standalone filter chain implementation. Which works both in synchronous and asynchronous modes.
It's possible to do following:
filterChain.add(new TransportFilter());
filterChain.add(new SSLFilter());
filterChain.add(new HttpFilter());
...........................
HttpRequest request = new HttpRequest(host, port, url);
Future writeFuture = filterChain.write(connection, HttpRequest()); Future readFuture = filterChain.read(connection);
HttpResponse response = (HttpResponse) readFuture.get(timeout, timeunit);
We're planning to have more interesting features: Stream filter, smart parser filter, HTTP support...
Posted at 07:53PM Oct 09, 2008 by oleksiys in Grizzly | Comments[0]
Grizzly 2.0 roadmap
- Status
- Expected release December 2008
- Tasks
- (required) Support TCP NIO transport DONE
- (required) Support UDP NIO transport
- (required) Support SSL
- (required) Performance. Grizzly 2.0 should be faster or equal to Grizzly 1.x
- (required) Documentation: complete javadoc for sources
- (required) Documentation: tutorial (set of blogs), which covers all common framework features
- (required) Support AIO's TCP/UDP as Transport
- (required) High level design of the http server module
- (feature) Standalone filter read/write DONE
- (feature) Stream filter
- (feature) Smart parser filter
Posted at 07:04PM Oct 09, 2008 by oleksiys in Grizzly | Comments[0]
Be careful with SelectionKey interests! :)
I spent one day catching the bug in the client side implementation of Grizzly 2.0
The code bellow:
selector.select();Set<SelectionKey> readyKeySet = selector.selectedKeys();
for(Iterator<SelectionKey> it = readyKeySet.iterator(); it.hasNext(); ) {
SelectionKey key = it.next();
....................
is usual way of processing I/O events with Java NIO.
The SelectionKey should now tell us, which operation is ready to be processed, but
int readyOperations = key.readyOps();
returns 0 8-)
This is really unexpected, because it means, that SelectionKey is not ready for any I/O operation, and even more...
selector.select() now returns immediately all the time with non-ready SelectionKey. So our Selector thread is loading CPU up-to 100%.
After one day of investigation and trying different patches I realized, that issue occurred, because I forgot to unregister OP_CONNECT interest from the SelectionKey. Once I added logic to unregister OP_CONNECT interest, when a channel gets connected - everything started to work just fine.
I saw such an issue just on my Mac, so not sure how other OS'es JVM will behave. Anyway if you don't want to spend the day catching strange NIO bugs - be careful with SelectionKey interests :)
Posted at 11:06AM Sep 24, 2008 by oleksiys in Grizzly | Comments[0]
Grizzly 2.0 is available on Maven
We started to push Grizzly 2.0 artifacts to the Maven repository [1].
Currently there are two Maven artifacts available: grizzly-framework[2] and grizzly-framework-samples[3]; which contain correspondently Grizzly 2.0 core classes and several samples.
Grizzly 2.0 sources could be retrieved using SVN [4] or online [5].
[1] http://download.java.net/maven/2
[2]
<groupId>org.glassfish.grizzly</groupId>
<artifactId>grizzly-framework</artifactId>
<version>2.0.0-SNAPSHOT</version>
[3]
<groupId>org.glassfish.grizzly.samples</groupId>
<artifactId>grizzly-samples</artifactId>
<version>2.0.0-SNAPSHOT</version>
[4] svn checkout https://www.dev.java.net/svn/grizzly/branches/2dot0
[5] https://grizzly.dev.java.net/source/browse/grizzly/branches/2dot0/
Posted at 04:38PM Sep 04, 2008 by oleksiys in Grizzly | Comments[3]
Starting new Grizzly 2.0
It's always so cool to start something new... Starting Grizzly 2.0 we decided to create new Grizzly from the very beginning, using our experience with Grizzly 1.x and community feedback we were getting during last years.
So until the end of this year 2008, we are planning to have completely new bear, which will be even more friendly and powerful :)
Everyone is welcome to take part in the discussion, how Grizzly 2.0 should look like and what should be improved comparing to Grizzly 1.x. For now we have some compiled draft-plan, which has initial ideas, design, examples.
Posted at 02:11PM Jun 20, 2008 by oleksiys in Grizzly | Comments[0]
Grizzly @ JavaOne 2008
Jeanfrancois and myself presented Grizzly @ JavaOne 2008 conference in San Francisco.
For me it was first presentation experience, so it wasn't perfect, but hope people who came and were interested in NIO and partially in Grizzly did get the information, what Grizzly is and how they can start to use it.
Here are our presentation slides. More information you can find on project Grizzly web site. And all your questions could be answered on our mailing lists.
Posted at 02:58PM May 14, 2008 by oleksiys in Grizzly | Comments[2]