[BUG] java.io.OptionalDataException error in OpenSearch 2.0 rc

Describe the bug
I am collecting data with OpenSearch with Logstash.
OpenSearch throws java.io.OptionalDataException error during service operation.
This did not happen in the OpenSearch 1.3 operating environment.
The occurrence of errors is irregular, and errors sometimes disappear after a certain period of time has passed.
Data collection is collected in different indexes depending on the data type, and some index collections work when an error occurs.
The data that is failing is not just a specific index, but a variety of things.
To Reproduce
Expected behavior
Plugins
Screenshots
Host/Environment (please complete the following information):

OS: CentOS Linux release 7.9.2009
Logstash: 7.16.2.
Version: OpenSearch works in docker, and images use opensearchproject/opensearch:1.3.0 / opensearchproject/opensearch:2.0.0-rc1

Additional context
[OpenSearch Log]
2022-05-18T23:44:42.965700981Z [2022-05-18T23:44:42,965][INFO ][o.o.j.s.JobSweeper       ] [91616f4bd951] Running full sweep
2022-05-18T23:49:42.966011507Z [2022-05-18T23:49:42,965][INFO ][o.o.j.s.JobSweeper       ] [91616f4bd951] Running full sweep
2022-05-18T23:49:43.072676353Z [2022-05-18T23:49:43,072][INFO ][o.o.a.t.CronTransportAction] [91616f4bd951] Start running AD hourly cron.
2022-05-18T23:49:43.072820561Z [2022-05-18T23:49:43,072][INFO ][o.o.a.t.ADTaskManager    ] [91616f4bd951] Start to maintain running historical tasks
2022-05-18T23:49:43.073109887Z [2022-05-18T23:49:43,073][INFO ][o.o.a.c.HourlyCron       ] [91616f4bd951] Hourly maintenance succeeds
2022-05-18T23:50:30.021645243Z [2022-05-18T23:50:30,012][WARN ][r.suppressed             ] [91616f4bd951] path: /indexname-*/_search, params: {index=indexname-*}
2022-05-18T23:50:30.021674167Z org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
2022-05-18T23:50:30.021678359Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:642) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021681538Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:360) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021684355Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:677) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021687051Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:457) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021689946Z 	at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:291) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021692560Z 	at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:72) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021695600Z 	at org.opensearch.transport.TransportService$6.handleException(TransportService.java:735) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021707976Z 	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:318) [opensearch-security-<IP_ADDRESS>-rc1.jar:<IP_ADDRESS>-rc1]
2022-05-18T23:50:30.021711627Z 	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1350) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021714318Z 	at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1459) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021717064Z 	at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1433) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021719769Z 	at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:967) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021723529Z 	at org.opensearch.transport.TransportService$3.sendRequest(TransportService.java:147) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021726036Z 	at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:869) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021728354Z 	at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:212) [opensearch-security-<IP_ADDRESS>-rc1.jar:<IP_ADDRESS>-rc1]
2022-05-18T23:50:30.021731148Z 	at org.opensearch.security.OpenSearchSecurityPlugin$7$2.sendRequest(OpenSearchSecurityPlugin.java:665) [opensearch-security-<IP_ADDRESS>-rc1.jar:<IP_ADDRESS>-rc1]
2022-05-18T23:50:30.021733902Z 	at org.opensearch.transport.TransportService.sendRequest(TransportService.java:756) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021736648Z 	at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:831) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021739309Z 	at org.opensearch.action.search.SearchTransportService.sendCanMatch(SearchTransportService.java:149) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021742052Z 	at org.opensearch.action.search.CanMatchPreFilterSearchPhase.executePhaseOnShard(CanMatchPreFilterSearchPhase.java:128) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021744771Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:278) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021747524Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:312) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021750314Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:249) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021753108Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:415) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021755926Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:215) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021758723Z 	at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:994) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021764118Z 	at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:757) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021767219Z 	at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:398) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021770070Z 	at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:78) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021772833Z 	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:136) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021775554Z 	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:101) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021778319Z 	at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021781263Z 	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:277) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021784064Z 	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:120) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021786868Z 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:194) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021789576Z 	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-<IP_ADDRESS>-rc1.jar:<IP_ADDRESS>-rc1]
2022-05-18T23:50:30.021792191Z 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021794631Z 	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-<IP_ADDRESS>-rc1.jar:<IP_ADDRESS>-rc1]
2022-05-18T23:50:30.021797236Z 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021800054Z 	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:325) [opensearch-security-<IP_ADDRESS>-rc1.jar:<IP_ADDRESS>-rc1]
2022-05-18T23:50:30.021802755Z 	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:157) [opensearch-security-<IP_ADDRESS>-rc1.jar:<IP_ADDRESS>-rc1]
2022-05-18T23:50:30.021806267Z 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021809178Z 	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021811919Z 	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021814537Z 	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021819342Z 	at org.opensearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:104) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021822194Z 	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:425) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021824698Z 	at org.opensearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:130) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021827375Z 	at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:123) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021830341Z 	at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:128) [opensearch-security-<IP_ADDRESS>-rc1.jar:<IP_ADDRESS>-rc1]
2022-05-18T23:50:30.021832662Z 	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:306) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021835200Z 	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:392) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021837529Z 	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:235) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021840258Z 	at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-<IP_ADDRESS>-rc1.jar:<IP_ADDRESS>-rc1]
2022-05-18T23:50:30.021843001Z 	at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:361) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021845584Z 	at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:440) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021848392Z 	at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:351) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021851677Z 	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021854379Z 	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021857188Z 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021859832Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021862547Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021865145Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021868012Z 	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021873155Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021876009Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021878647Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021881288Z 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021884253Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021887006Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021890227Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021893150Z 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021895772Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021898247Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021900791Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021903611Z 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021906357Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021909121Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021911671Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021914332Z 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021917065Z 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021919797Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021924273Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021927154Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021930364Z 	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021933214Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021935825Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021938494Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021941185Z 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021943772Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021946507Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021949096Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021951894Z 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021954600Z 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021957283Z 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021959941Z 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021962822Z 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021965687Z 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021968924Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021971771Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021974631Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021979112Z 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021982181Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021984750Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021987493Z 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021990170Z 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021992886Z 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021995570Z 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021998068Z 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.022005132Z 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.022007714Z 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.022010101Z 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.022013231Z 	at java.lang.Thread.run(Thread.java:833) [?:?]
2022-05-18T23:50:30.022015580Z Caused by: org.opensearch.OpenSearchException: java.io.OptionalDataException
2022-05-18T23:50:30.022020352Z 	at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:185) ~[?:?]
2022-05-18T23:50:30.022023058Z 	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:155) ~[?:?]
2022-05-18T23:50:30.022025907Z 	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:97) ~[?:?]
2022-05-18T23:50:30.022028556Z 	at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
2022-05-18T23:50:30.022031217Z 	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
2022-05-18T23:50:30.022033939Z 	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
2022-05-18T23:50:30.022036620Z 	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:98) ~[opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.022042345Z 	at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:931) ~[opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.022045459Z 	... 101 more
2022-05-18T23:50:30.022047895Z Caused by: java.io.OptionalDataException
2022-05-18T23:50:30.022050506Z 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1767) ~[?:?]
2022-05-18T23:50:30.022053172Z 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:514) ~[?:?]
2022-05-18T23:50:30.022055439Z 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:472) ~[?:?]
2022-05-18T23:50:30.022057905Z 	at java.util.HashSet.readObject(HashSet.java:345) ~[?:?]
2022-05-18T23:50:30.022060408Z 	at jdk.internal.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) ~[?:?]
2022-05-18T23:50:30.022062944Z 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2022-05-18T23:50:30.022065516Z 	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
2022-05-18T23:50:30.022068083Z 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1231) ~[?:?]
2022-05-18T23:50:30.022070808Z 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2434) ~[?:?]
2022-05-18T23:50:30.022073271Z 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2268) ~[?:?]
2022-05-18T23:50:30.022075787Z 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744) ~[?:?]
2022-05-18T23:50:30.022078237Z 	at java.io.ObjectInputStream$FieldValues.<init>(ObjectInputStream.java:2617) ~[?:?]
2022-05-18T23:50:30.022081688Z 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2468) ~[?:?]
2022-05-18T23:50:30.022087904Z 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2268) ~[?:?]
2022-05-18T23:50:30.022090864Z 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744) ~[?:?]
2022-05-18T23:50:30.022093895Z 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:514) ~[?:?]
2022-05-18T23:50:30.022096278Z 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:472) ~[?:?]
2022-05-18T23:50:30.022098911Z 	at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:183) ~[?:?]
2022-05-18T23:50:30.022101744Z 	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:155) ~[?:?]
2022-05-18T23:50:30.022104165Z 	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:97) ~[?:?]
2022-05-18T23:50:30.022106691Z 	at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
2022-05-18T23:50:30.022109481Z 	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
2022-05-18T23:50:30.022112185Z 	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
2022-05-18T23:50:30.022117725Z 	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:98) ~[opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.022119699Z 	at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:931) ~[opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.022121470Z 	... 101 more

@adnapibar ^ says The occurrence of errors is irregular, and errors sometimes disappear after a certain period of time has passed.

@dblock Yes agree! But I think we need more details that can help us to reproduce this error.  Also, does this error happen with other API calls during the time? I see one discussion on es 7.9 reporting similar issue (not with logstash) https://discuss.elastic.co/t/temp-downtimes-optionaldataexception-500-on-all-apis-aws/257571 which may be completely unrelated.

Thank you for your interest.
Our service collects logs with logstash, stores them with ElasticSearch, and retrieves logs with a NodeJS-based web server.
We are changing from using ElasticSearch (with or without the Opendistro plugin) service to OpenSearch service.
logstash uses OSS version 7.16.2 and the output plugin uses logstash-output-elasticsearch (11.2.3).
The web server uses the javascript client elastic/elasticsearch.

As far as I know, the OptionalDataException error has been confirmed so far in the OpenSearch V2.0Rc environment.
Based on the symptoms occurring in the test environment, it seems to occur as follows:

OptionalDataException seems to be thrown first in Logstash during log collection.

It also happens through the OpenSearch search API. (Occurs when a request is made from the web server to the OpenSearch server)

Logs are collected in several types, and the log types in which OptionalDataException occurs are various. However, it seems that the object type is included in the data mapping among the collection histories in each log. For example, in the logindex generated by month, there was Object type mapping in logindex-April, and there was no Objecty type mapping in logindex-May, and it also occurred in the process of inserting data without object type.

In the existing OpenSearch or ElasticSearch, the vm.max_map_count value was not set separately. (It seems that the default value of 65,530 was applied)
After the occurrence of OptionalDataException was confirmed in the OpenSearch 2.0 Rc1 environment, when this value was changed to the recommended value (262,144), it seems that OptionalDataException occurs less.

After the vm.max_map_count value was changed, I did not experience the symptom that an OptionalDataException error occurred when searching in the web server.

When an OptionalDataException occurred before the vm.max_map_count value was changed, there were cases where the login function could not be used even in OpenSearchDashboard.
(500 status code, Internal Server Error)

When an OptionalDataException occurred before the vm.max_map_count value was changed, the symptom seemed to continue until the OpenSearch service was restarted.

After changing the value of vm.max_map_count, I monitored the occurrence of OptionalDataException (only in Logstash, not when looking up the web server). It occurs about 5 times per minute for a certain period of time, then does not occur for a certain period of time and then occurs irregularly.

In general, when _cat/indices is searched, the response is in the following form.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open indexname JermuzKkTAa0dL7AxZ6l3A 1 0 449683 0 418.6mb 418.6mb

When an OptionalDataException occurs in Logstash, when _cat/indices is searched with OpenSearch through curl, document and size information may not appear for all indexes as follows.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open indexname JermuzKkTAa0dL7AxZ6l3A
I wonder how many cases of this symptom are there, and I wonder if it is an opensearch bug, or a problem with the version of the library we use or a problem with data structure.
So far, it seems difficult to predict the cause of the occurrence.
Obviously, I have not experienced it with any version other than OpenSearch 2.0Rc1.

Thank you for your interest.
Our service collects logs with logstash, stores them with ElasticSearch, and retrieves logs with a NodeJS-based web server.
We are changing from using ElasticSearch (with or without the Opendistro plugin) service to OpenSearch service.
logstash uses OSS version 7.16.2 and the output plugin uses logstash-output-opensearch (1.2.0).
The web server uses the javascript client elastic/elasticsearch.

As far as I know, the OptionalDataException error has been confirmed so far in the OpenSearch V2.0Rc environment.
Based on the symptoms occurring in the test environment, it seems to occur as follows:

OptionalDataException seems to be thrown first in Logstash during log collection.

It also happens through the OpenSearch search API. (Occurs when a request is made from the web server to the OpenSearch server)

Logs are collected in several types, and the log types in which OptionalDataException occurs are various. However, it seems that the object type is included in the data mapping among the collection histories in each log. For example, in the logindex generated by month, there was Object type mapping in logindex-April, and there was no Objecty type mapping in logindex-May, and it also occurred in the process of inserting data without object type. (The important thing is that OptionalDataException is thrown randomly.)

In the existing OpenSearch or ElasticSearch, the vm.max_map_count value was not set separately. (It seems that the default value of 65,530 was applied)
After the occurrence of OptionalDataException was confirmed in the OpenSearch 2.0 Rc1 environment, when this value was changed to the recommended value (262,144), it seems that OptionalDataException occurs less.

After the vm.max_map_count value was changed, I did not experience the symptom that an OptionalDataException error occurred when searching in the web server.

When an OptionalDataException occurred before the vm.max_map_count value was changed, there were cases where the login function could not be used even in OpenSearchDashboard.
(500 status code, Internal Server Error)

When an OptionalDataException occurred before the vm.max_map_count value was changed, the symptom seemed to continue until the OpenSearch service was restarted.

After changing the value of vm.max_map_count, I monitored the occurrence of OptionalDataException (only in Logstash, not when looking up the web server). It occurs about 5 times per minute for a certain period of time, then does not occur for a certain period of time and then occurs irregularly.

In general, when _cat/indices is searched, the response is in the following form.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open indexname JermuzKkTAa0dL7AxZ6l3A 1 0 449683 0 418.6mb 418.6mb
When an OptionalDataException occurs in Logstash, when _cat/indices is searched with OpenSearch through curl, document and size information may not appear for all indexes as follows.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open indexname JermuzKkTAa0dL7AxZ6l3A

I wonder how many cases of this symptom are there, and I wonder if it is an opensearch bug, or a problem with the version of the library we use or a problem with data structure.
So far, it seems difficult to predict the cause of the occurrence.
Obviously, I have not experienced it with any version other than OpenSearch 2.0Rc1.

It happened with 2.0.0 official release too. I migrated my cluster on 8h30 and the problem started 9h50 and took 1h do stop. It recovered from the error alone (no intervention).
It is critical for us and now it is difficult to revert my upgrade.

The same symptoms appear in the full version of Opensearch 2.0.
The environment file used for testing is attached.
ostest.tar.gz
It still happens as a result of checking for the following:

After the vm.max_map_count value was changed, I did not experience the symptom that an OptionalDataException error occurred when searching in the web server.

After all the services are up (docker-compose up -d), an OptionalDataException is thrown at some point after a certain amount of time has passed. (Occurs within 1 hour, occurs after 10 hours, etc.)
The peculiar thing is that when monitoring with only the service running, OptionalDataException occurs in Logstash, but the log is not visible in OpenSearch.
At this time, when accessing the OpenSearchDashboard UI, an error 500 occurs, and an OptionalDataEception occurs in the OpenSearchDashboard log and OpenSearch log.

Hi @HyunSeaHoon Thank you so much for providing such detailed steps to reproduce, appreciate your time.
And thanks @louzadod for reporting the issue as well.
I will discuss the issue with the other maintainers, and take it as a high priority.

There are breaking changes in the bulk API. Probably Logstash Output Plugin needs some adjustment.

The architecture we adopted is as follows:
[Filebeat] --> [Logstash] --> [OS Client Nodes] --> [OS Data Nodes]
When the problem happens, the logs from Logstash to OpenSearch Client Nodes stops flowing. I assumed the problem was on Logstash and tried to restart it. After LS restart, logs were not flowing again.
Only when I restarted the OS Client Nodes, the logs started to flow again. It looks like a problem in the OpenSearch side.

Hi! Any update on this one?
I've got the same problem after upgrading from 1.2.4 to 2.0.0. I've got 2 data nodes and 1 master node. Several times during a day this stuff happens on random node. Cant read or write to this node until restart. Also no related warning or errors in +-5 minutes period.
Writing data directly to data nodes with fluent-bit (opensearch input), reading with grafana and opensearch-dashboards 2.0.0. No problems occurred before upgrade

Same happens here. Multiple times a day Logstash fails to sent logs to OpenSearch cluster. Started happening after upgrade to V2.
Is downgrading back to v1.3.2 an option? maybe some other workaround?

The only workaround that is helping me a lot is to schedule restart hourly for the client nodes.

Yes as temp workaround we run 'DELETE _plugins/_security/api/cache' this solves the problem till next cache expire.

Experienced the exact same error on a OS 2.0 cluster without Logstash. It occurred after indexing close to 1 billion documents with elasticsearch python client (7.10.0). Same exact python code didn't show the error on 1.3 OS and also the error doesn't happen consistently on OS 2.0 either.

I will be looking into this.

Upgraded to version 2.0.1 on Friday and turned off all scheduled cache purge. There were no issues for some time but on Sunday got same error again. So issue still exists but appears less frequently.

Hi @JustinasKO . I also tried 2.0.1 and confirmed that it does not fix this issue.

Same here, latest fix didn't seem to have any impact. Still get a node randomly failing which causes this error. In that case I have to restart the node as a tmp fix.
I have installed it via helm chart if it means anything

While making changes in the security information (user, roles and so on) via OS Dashboards, the problem appeared again.
I had to restart again my clients do get back to normal promptly. One possible cause may be the security plugin.

I'm also trying another workaround from my side. There were new internal users that were missing in the skip_users section of config.yml (security).
The implication was the following exception on client nodes:
org.opensearch.OpenSearchSecurityException: OpenSearchSecurityException[No user someuser found]

I updated this information and no exception being raised. Let's see what happens.

d the problem started 9h50 and took 1h do stop. It recovered from the error alone (no intervention) and I don't know if it is going to come back.
It is critical for us and now it is difficult to revert my upgrade.

My problem is almost 3-4 times a day at hh:50 or hh:51 if it could help to identify!

I suspect this PR to be the culprit, but I don't know enough to pinpoint the problem: https://github.com/opensearch-project/security/commit/ef0d40a339ef651fce8a4e44b1af39eca765c438

The assumption I wrote above is wrong. I removed all org.opensearch.OpenSearchSecurityException and the problem keeps happening.
Here are my current actions now:

increased cache.ttl_minutes to 6 hours
daily cache flush

@tlfeng , one important difference about the way you ingest data and logstash in my case: logstash uses bulk api.
I hope it helps. Regards.

Hi @louzadod Thank you so much for providing the important information of using "Bulk API"!
I also ask @amitgalitz for detail, he mentioned that he used bulk API as well, with python client. So I will have a try today.

I am also using the Bulk API with the elasticsearch-net client.  I followed the advice above and changed the
ttl in my opensearch.yml config to plugins.security.cache.ttl_minutes: 1440.
This decreased the frequency of errors drastically.  I then implemented logic to detect the error and call the flush cache api in my app before retrying. It's super hacky but it's getting me by for now.
if (item.Error.Reason == "java.io.OptionalDataException")
{
    try
    {
        var flushCacheResponse = await _simpleCacheClient.FlushAuthCacheAsync();
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Error flushing auth cache.");
    }
}

   public class SimpleCacheClient : ISimpleCacheClient
   {
       private readonly HttpClient _http;
       private readonly ILogger<SimpleCacheClient> _logger;
       private readonly AppOptions _options;

       public SimpleCacheClient(
           HttpClient httpClient,
           IOptions<AppOptions> options,
           ILogger<SimpleCacheClient> logger
           )
       {
           _http = httpClient;
           _logger = logger;
           _options = options.Value;
       }

       public async Task<FlushCacheResponse> FlushAuthCacheAsync()
       {
           _logger.LogDebug("Begin {method}", nameof(FlushAuthCacheAsync));

           List<Exception> innerExceptions = null;
           foreach (var item in _options.ElasticClusterNodes)
           {
               try
               {
                   var url = $"{item}/_opendistro/_security/api/cache";
                   var response = await _http.DeleteAsync(url);
                   _logger.LogDebug("Attempting to flush cache at url {url}", url);

                   if (response.IsSuccessStatusCode)
                   {
                       using var stream = await response.Content.ReadAsStreamAsync();
                       return JsonSerializer.Deserialize<FlushCacheResponse>(stream);
                   }
                   else
                   {
                       throw new HttpRequestException($"Error sending request: {response.ReasonPhrase}");
                   }
               }
               catch (Exception ex)
               {
                   if (innerExceptions == null)
                       innerExceptions = new List<Exception>();

                   innerExceptions.Add(ex);
               }
               await Task.Delay(1000);
           }

           var finalException = new AggregateException("Unable to flush the cache from any of the available OpenSearch hosts.", innerExceptions);
           throw finalException;
       }
   }

Right now everyone, who is using OpenSearch 2.x (we are running OpenSearch 2.0.1) with Security Plugin and Logstash 7.16.2 with OpenSearch output plugin, is probably losing data. And because it happens sporadic, it could go completely unnoticed.

So I'm beginning to think that this may be caused by an incompatibility with the Elasticsearch version of the clients.  I implemented my own bulk index client and haven't experienced these issues.
Originally I was using elasticsearch-net client version 7.12.  I'm not using Logstash but I imagine it is using the Elasticsearch versions of the client libraries internally and is running into the same issues.
I think we have just hit a point where the API has diverged enough that the Elasticsearch clients are no longer compatible with > 2.x OpenSearch.

I can confirm, it solved my issue with: plugins.security.cache.ttl_minutes: 1440 in opensearch.yml!

So I'm beginning to think that this may be caused by an incompatibility with the Elasticsearch version of the clients. I implemented my own bulk index client and haven't experienced these issues.
Originally I was using elasticsearch-net client version 7.12. I'm not using Logstash but I imagine it is using the Elasticsearch versions of the client libraries internally and is running into the same issues.
I think we have just hit a point where the API has diverged enough that the Elasticsearch clients are no longer compatible with > 2.x OpenSearch.

@msoler8785  I do remember seeing this change in OpenSearch 2.

I ran the exact same python script (repeatedly calling bulk api) once utilizing elasticsearch-py (7.10.0) and once using openearch-py client (2.0.0). On the first attempt (elasticsearch client) to index 1b docs (~111gb), data went missing and I got the same exceptions as mentioned above with only 1b docs (~105gb) indexed. Using the opensearch-py client the exception didn't occur. Since so far this exception seems to be somewhat random, I can't say that with more data the opensearch client wont experience the same issue but seems to likely be related to incompatibility with elasticsearch client

We are using 2.0.0, and are also using the Java client 2.0.0 (opensearch-rest-client and opensearch-rest-high-level-client) and have been experiencing the same issues as others on this thread. So not sure it is a client incompatibility. As with others happens intermittently. We are running a 3 node cluster, and problem may occur only once in a few days or within hours.
Caused by: org.opensearch.OpenSearchException: java.io.OptionalDataException
        at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:185) ~[?:?]
       ...
Caused by: java.io.IOException
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1767) ~[?:?]

I can confirm, it solved my issue with: plugins.security.cache.ttl_minutes: 1440 in opensearch.yml!

Indeed, I solved the issue as well, when I set this configuration!

Has anyone with this problem updated to 2.1.0 and verified if it still happens?
Adding ttl_minutes does not solve the problem, it just alleviates it.

We are still having this problem with 2.1.0.
Looks like it happens after each TTL timeout as mentioned here: https://github.com/opensearch-project/security/issues/1927.

I'm still trying to reproduce to bug locally, and hope it can help me locate the problem.
From the above discussions, I think the key to reproduce the bug is sending bulk API request with a client.
Based on the feedback from @w39hh (https://github.com/opensearch-project/OpenSearch/issues/3395#issuecomment-1176549127), I wrote a sample code to use Java high level rest client 2.0.0 to send bulk API request.
But I haven't seen the OptionalDataException yet. 🙁
public static void main(String[] args) throws IOException, InterruptedException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
        String timeStamp = new SimpleDateFormat("yyyyMMdd_HHmmss").format(Calendar.getInstance().getTime());

        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY,
                new UsernamePasswordCredentials("admin", "admin"));

        SSLContext sslContext = new SSLContextBuilder()
                .loadTrustMaterial(null, (certificate, authType) -> true).build();

        RestClientBuilder builder = RestClient.builder(new HttpHost("localhost", 9200, "https"))
                .setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
                    @Override
                    public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                        return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
                                .setSSLContext(sslContext)
                                .setSSLHostnameVerifier(new NoopHostnameVerifier());
                    }
                });

        RestHighLevelClient client = new RestHighLevelClient(builder);

        for(int i=1110; i<19000; i++) {
            BulkRequest request = new BulkRequest();
            request.add(new IndexRequest(String.valueOf(i)).id("1")
                    .source(XContentType.JSON,"date", timeStamp));
            request.add(new DeleteRequest(String.valueOf(i), "1"));
            request.add(new IndexRequest(String.valueOf(i)).id("2")
                    .source(XContentType.JSON,"date", timeStamp));
            request.add(new UpdateRequest(String.valueOf(i), "2")
                    .doc(XContentType.JSON,"field2", "value2"));
            BulkResponse bulkResponse = client.bulk(request, RequestOptions.DEFAULT);
            System.out.println(i + " " + timeStamp);
            System.out.println(bulkResponse.status());
            TimeUnit.SECONDS.sleep(1);
        }
        client.close();
    }

In pom.xml
    <dependencies>
        <dependency>
            <groupId>org.opensearch.client</groupId>
            <artifactId>opensearch-rest-high-level-client</artifactId>
            <version>2.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.18.0</version>
        </dependency>
    </dependencies>

@tlfeng When the issue occurs for us, it is not related to a Bulk API request.  We have applied the mitigations

plugins.security.cache.ttl_minutes: 1440
restart every 3 hours

But we still get the issue, although much less frequently, but typically once a day.  I have added a much larger portion of the stack trace in the event it helps. (**** = removed text)
[2022-07-14T17:53:20,923][WARN ][r.suppressed             ] [<IP_ADDRESS>] path: /****-assign/_search, params: {typed_keys=true, max_concurrent_shard_requests=5, ignore_unavailable=false, expand_wildcards=open, allow_no_indices=true, ignore_throttled=true, index=dawn-assign, search_type=query_then_fetch, batched_reduce_size=512, ccs_minimize_roundtrips=true}
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:644) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:362) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:679) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:459) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:293) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:74) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService$6.handleException(TransportService.java:750) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:318) [opensearch-security-<IP_ADDRESS>.jar:<IP_ADDRESS>]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1370) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:301) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.handleException(InboundHandler.java:418) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:410) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:158) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) [transport-netty4-client-2.0.0.jar:2.0.0]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
Caused by: org.opensearch.OpenSearchException: java.io.OptionalDataException
        at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:185) ~[?:?]
        at org.opensearch.security.auditlog.impl.AbstractAuditLog.getUser(AbstractAuditLog.java:597) ~[?:?]
        at org.opensearch.security.auditlog.impl.AbstractAuditLog.logMissingPrivileges(AbstractAuditLog.java:202) ~[?:?]
        at org.opensearch.security.auditlog.impl.AuditLogImpl.logMissingPrivileges(AuditLogImpl.java:162) ~[?:?]
        at org.opensearch.security.auditlog.AuditLogSslExceptionHandler.logError(AuditLogSslExceptionHandler.java:77) ~[?:?]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:169) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:103) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:249) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:132) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]

Caused by: java.io.IOException
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1767) ~[?:?]
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:514) ~[?:?]
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:472) ~[?:?]
        at java.util.HashSet.readObject(HashSet.java:345) ~[?:?]
        at jdk.internal.reflect.GeneratedMethodAccessor47.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1231) ~[?:?]
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2434) ~[?:?]
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2268) ~[?:?]
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744) ~[?:?]
        at java.io.ObjectInputStream$FieldValues.<init>(ObjectInputStream.java:2617) ~[?:?]
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2468) ~[?:?]
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2268) ~[?:?]
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744) ~[?:?]
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:514) ~[?:?]
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:472) ~[?:?]
        at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:183) ~[?:?]
        at org.opensearch.security.auditlog.impl.AbstractAuditLog.getUser(AbstractAuditLog.java:597) ~[?:?]
        at org.opensearch.security.auditlog.impl.AbstractAuditLog.logMissingPrivileges(AbstractAuditLog.java:202) ~[?:?]
        at org.opensearch.security.auditlog.impl.AuditLogImpl.logMissingPrivileges(AuditLogImpl.java:162) ~[?:?]
        at org.opensearch.security.auditlog.AuditLogSslExceptionHandler.logError(AuditLogSslExceptionHandler.java:77) ~[?:?]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:169) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:103) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:249) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:132) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]

Faced same issue after upgrade from 1.3
so, I've found how to reproduce this bug locally
I used vm (4cpu, hdd)
cluster was created by official docker compose file

create cluster

curl https://opensearch.org/samples/docker-compose.yml -O
docker-compose up

run fill.sh script to send sample data to the index

#!/bin/sh
while true
  do
    curl -XPOST -k -u admin:admin https://localhost:9200/test-index/_doc \
      -H "Content-Type: application/json" -d @test_data.json
  done

run in infinity loop on  purge cache url

while true;do curl -k -u admin:admin -X DELETE https://localhost:9200/_plugins/_security/api/cache; done

run in infinity loop  on _nodes stats

while true; do curl -s -k -u admin:admin https://localhost:9200/_nodes -o /dev/null; done

4 java.io.OptionalDataException will appear in minutes (on my env :)
sh script and json payload can be found in my repo https://github.com/denisvll/os_test
docker ps
CONTAINER ID   IMAGE                                            COMMAND                  CREATED         STATUS         PORTS                                                                                NAMES
f9f8f58d95bb   opensearchproject/opensearch:latest              "./opensearch-docker…"   3 minutes ago   Up 7 seconds   <IP_ADDRESS>:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp, <IP_ADDRESS>:9600->9600/tcp, :::9600->9600/tcp, 9650/tcp   opensearch-node1
5fd8e3c80116   opensearchproject/opensearch-dashboards:latest   "./opensearch-dashbo…"   3 minutes ago   Up 7 seconds   <IP_ADDRESS>:5601->5601/tcp, :::5601->5601/tcp                                                                  opensearch-dashboards
523714cde06c   opensearchproject/opensearch:latest              "./opensearch-docker…"   3 minutes ago   Up 7 seconds   9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp                                                                     opensearch-node2

docker images
REPOSITORY                                TAG       IMAGE ID       CREATED       SIZE
opensearchproject/opensearch              latest    9a3e759bec77   2 weeks ago   854MB
opensearchproject/opensearch-dashboards   latest    7da32241e476   2 weeks ago   998MB

The bug described in this issue is also happening to me.
My OpenSearch infra context is:

Component
Version

OpenSearch
2.0.1

Amazon Linux
2

Logstash
7.16.2 (logstash-oss-with-opensearch-output-plugin)

My workaround solution was to create a script that checks the log ingestion status. If the ingestion log has no activity in the last 1 minute, it is executed DELETE _plugins/_security/api/cache in OpenSearch API. This script run over Linux Cron and is executed for every 5 minutes.
The following query is to check log ingestion is stopped:
GET my-index*/
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "ingest_time": {
              "from": "now-1m",
              "to": "now"
            }
          }
        }
      ]
    }
  }
}

This is also happening with me.


Support for project level node_modules directory?

I have a question regarding the placement of my node_modules directory. Is is possible to place a node_modules directory at the project.json level and have it accessible within function definitions? If so, will apex deploy handle the inclusion of these into the pack before deploying?

You can use symlinks: http://apex.run/#structuring-projects

What would you gain by this? The npm way to do things would be to have these dependencies in each functions node_modules. If they're local then npm symlinks for you anyways.

Up has the same problem :'( seems like mono repos utilize Node's parent node_modules resolution stuff instead of symlinks, so Apex & Up would have to be Node-aware


feat(agenda): fallback link to plaintext version

Adds fallback links to loading screen and error screen:

Fixes #4655

Nice fix. I just wish we wouldn't continue to build a ton of custom CSS instead of using b5. It will be a pain to move to newer bs versions.


Added solution to Cricket Ranking question from codechef

Have you read the Contributing Guidelines on Pull Requests?
yes
Description
Added solution to Cricket Ranking question from codechef July starters in C language
Checklist

[x] I've read the contribution guidelines.
[x] I've checked the issue list before deciding what to submit.

Related Issues or Pull Requests
#761
Fill up the DCP form
If you are a part of DCP 21, fill up this form after the PR is merged.
The links provided in the form will be checked thoroughly, if someone tries fooling around, strict actions will be taken against the individual.

Cool !


[Storage] Hard deprecate old download APIs

There are 4 APIs in Storage that has been "soft deprecated" like this:
https://github.com/Azure/azure-sdk-for-java/blob/1da4fdf5222226a66381c691cf2d8ae7dd32242c/sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/blob/specialized/BlobClientBase.java#L725-L735
The APIs are:

BlobClientBase.download(OutputStream stream)
BlobClientBase.downloadWithResponse(OutputStream stream, BlobRange range,       DownloadRetryOptions options, BlobRequestConditions requestConditions, boolean getRangeContentMd5,       Duration timeout, Context context)
BlobAsyncClientBase.download()
BlobAsyncClientBase.downloadWithResponse(BlobRange range, DownloadRetryOptions options,       BlobRequestConditions requestConditions, boolean getRangeContentMd5)

These APIs should:

Be hard marked with @Deprecated
Javadoc warning should be converted into @deprecated section
Samples using deprecated APIs must be updated

PR was merged


[BUG]-DM USB Input - Input routing not selecting the correct card

Was this bug identified in a specific build version?
1.10.0
Describe the bug
The outputSelector value is being used to determine the input card value instead of the input selector value
Offending code:
if ((sigType & eRoutingSignalType.UsbInput) == eRoutingSignalType.UsbInput)
            {
                Debug.Console(2, this, "Executing USB Input switch.\r\n in:{0} output: {1}", inputSelector, outputSelector);
                if (outputSelector > chassisSize)
                {
                    uint outputIndex;

                    if (chassisSize == 8)
                    {
                        outputIndex = (uint) inputSelector - 16;
                    }
                    else
                    {
                        outputIndex = inputSelector - chassisSize;
                    }
                    dmCard = Chassis.Outputs[outputIndex];
                }
                else
                {
                    dmCard = Chassis.Inputs[inputSelector];
                }

                ExecuteSwitch(dmCard, Chassis.Inputs[outputSelector], sigType);
                return;
            }

@andrew-welker Has this been fixed?


[WIP] Add support for HuggingFace Chat Templates

Hey there, I revised my prior PR to instead implement the HF Chat Template support: https://huggingface.co/docs/transformers/main/en/chat_templating
This PR has support for tokenizers which have added a template, and also supports specifying a jinja2 template file through command-line arguments for models which haven't added the template to the tokenizer, or if a user wanted to modify the template.

Well, I somehow just messed up this branch. Will get that fixed.
Also got some feedback on Discord to remove FastChat dependency.

Closed, and replaced by: https://github.com/vllm-project/vllm/pull/1493


Add json output for CPython tests

Related to #1671

See #1838 (example section):
This pr implements the code object co_varnames attribute, but the value differs from the ones cpython and pypy present...

Does it produce the result in real-time, or only after the processing is finished?
Is it a custom format, or something like https://chromium.googlesource.com/chromium/src.git/+/master/docs/testing/json_test_results_format.md was used for an inspiration?

@abitrolly I'm not sure what you mean by your first question, but as to your second I think it's basically just the data that unittest outputs itself, converted to JSON. I didn't write the runner myself, I modified it from https://github.com/agramian/custom-text-test-runner.

@ep12 I'll look into fixing co_varnames separately.


[Question/possible enhancement suggestion] - How much radio silence before a device is considered to be unavailable?

I am working on adding BTHome support to a device and in testing ran into a question that - initially - I couldn't find a clear answer for.
While trying to answer the question, it sorta morphed into an idea that might make the protocol a bit more power efficient and/or more responsive for some use-cases.

For a device that has the BTHome Device Information byte set to 0x40 (not encrypted, not trigger based, version 2), How long from last transmission until the device is considered unavailable?
Since the BTHome protocol definition doesn't explicitly define the timing for bit 2, i'm assuming that the policy for unavailable is up to each implementation.
In testing with ESPHome/BT-Proxy -> Home Assistant, this appears to be 5 min/300 seconds.
This 300 second cutoff appears to be defined here.
This leads me to my (revised) question / suggestion:
Is there value in adding a "consider me unavailable If you don't hear another advertisement from me by $secondsFromNow" field to the advertisement payload?
The use case I had in mind: being able to tune the sensitivity of presence detection instead of the binary "it will check in 12 times an hour or ... whenever" choice.
In my current use-case, it would be nice to trigger actions no more than 60 seconds after a device stops phoning home.
Example: a beacon device attached to a pet collar. An escaped animal could make it quite far from the receiver before the 300s window elapses!
But the inverse is also true: devices that have a regular check-in interval measured in tens of min/hours are either forced to check in every 300 seconds to avoid going offline or set the "check in ... whenever" bit which means they'll (almost) never appear as offline... even if they are offline.
E.G.: a heating fuel tank level sensor. During the summer, phoning home every few hours is more than enough to confirm that the sensor is still alive as the fuel level will be incredibly stable. During the winter the fuel level might drop measurably every hour or so.
If the fuel measure device sets the "irregular interval" bit then there's nothing to notify the user that the device failed a scheduled liveliness check.
If the fuel measure device sets the "regular interval" bit than it needs to wake up at least 288 times per day just to report that ... there's still no measurable change in fuel level. Any less often and the user risks getting a false "device offline" notification from HA.
If the fuel measure device sets a "consider me offline if no update by 43200 seconds from now" byte in it's advertisement then HA will still expect a check-in 2x/day or more often.
During the winter, the sensor might be reporting fuel level changes hourly so this liveliness interval is being met.
During the summer, waking up 2x a day for liveliness is adequate.
If the sensor fails for some reason, the missed liveliness check will result in an appropriate "device offline" status in a timely manner.

The unavailable timeout in Home Assistant is indeed set by the Bluetooth integration, not by the BTHome integration in HA, and also can’t be set in the BTHome protocol itself. It might not be needed to set it in the protocol, as you can also decide to implement a device specific configuration option that sets the time out (if you want to use something else than the default HA Bluetooth timeout).

as you can also decide to implement a device specific configuration option that sets the time out (if you want to use something else than the default HA Bluetooth timeout).

Where/How is this done?

That would require adding a config option in the Home Assistant BTHome integration, which is part of the home assistant core repository. It might also require some changes to the Bluetooth integration. It's not something I have time for unfortunately

It might also require some changes to the Bluetooth integration.

I misunderstood. Presently it's not possible.
I assumed this might be the case; the whole HA component would likely need a re-factor to keep track of the per-entity (or would you implement it per PHY MAC?!) timeouts.
Adding manual user/configurable timeouts per entity would be the first (and probably biggest?) step. Then adding a new ID and a few bytes for the timeout would be a relatively simple addition to the actual BT-Home protocol.


Add header in flat binary files

Having the dtype and shape in the files will make the cluster store code much simpler.

dtype and shape as ascii string in a fixed-size header
Remove the need to specify the dtype and shape explicitly in the store and in cluster stats

Example of a header:
PHY FORMAT V1
dtype=float64
shape=(12,4,12)  [padded with spaces]
BINARYBLOB
(64 bytes header)

The alternative would be to use .npy but I think there are bugs between Python 2 and 3.

You should ask @kdharris101 about this. I get the impression you mean that this would be for "internal use only" files, but if you use something more generic - specifically, like a JSON metadata file alongside the flat binary - then this achieves the same thing as what you suggest but improves readability, compatibility, and perhaps becomes the seed for a better file format down the road. It's probably worth making these decisions carefully now.

very good point, I'm currently writing an e-mail specifically about this issue!

e-mail sent...

@nippoo I'm now using .npy for spikedetekt's cache: it's such a pain to keep track of the shape/dtype in the code. An alternative might be to use an external metadata file in JSON or YAML. But for now it's just simpler to use .npy (even if might be less reliable and slower).
It would be better to use the same format for the cluster store. But, this format should probably also be the same than the future kwik format, for consistency. So we'll need to settle on a single format at some point...

Cluster store really doesn't need to be the same as the central format: it's just a cache, and worst-case will take 20 minutes to regenerate.
I suggest we use .npy. Is it slower? I'd be very surprised. If it's slow then we should just write a loader ourselves. It's just a flat binary with a header.

yes it's just a cache, but dealing with several slightly different formats in a same program is just more painful (more code, more complexity, etc.).
can you make quick benchmarks with npy (small and large arrays)? then we'll know for sure if it's a problem or not

npy vs flat binary?

yes (for flat binary, you'll need to give the dtype and shape in the code)

just test slicing / copying the array or something?

generate a NumPy array, save it in npy, and in flat binary. Then load both and timeit that. But you may need to take care about the OS cache...

we'll use NPY


Fix for the disabled effect of Button component.

Changes proposed

Added disabled effect for clear/reset button when no tags are selected.
Made some css tweaks for the alignment of tags and it's numbers.

Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[ ] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[ ] All new and existing tests passed.
[x] This PR does not contain plagiarized content.
[x] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

Can you please add a screenshot?

Can you please add a screenshot?

Before:

After:

There seems to be multiple changes/fixes in 1 PR, please only do the fix not other changes

Fix is one only, others changes are an attempt to improve the alignment of the tag numbers. Anyways, if you don't want I will undo those changes.

Those extra changes have broken the design in other places. Yes best to remove them so the PR is not blocked by additional changes.
Some examples of the knock on effects...

Thank you for fixing 👍

Most welcome 😇


Made ability to add option attributes for the Select element.

Option attributes is very important feature for Select element. I have added it to one element. Now, we can do follow:
echo (new Select('age', ['10' => 'Teen', 20 => 'Twenty'])->setOptAttributes([10 => ['data-key' => 'some-key']]);

and we got result:
<select name="age">
    <option value="10" data-key="some-key">Teen</option>
    <option value="20">Twenty</option>
</select>

It's beaniful, isn't it? =)

Coverage increased (+0.2%) to 95.248% when pulling 5933c7ac2b0c36b6cd1f518598590e5ac93c5779 on bupy7:select-opt-attribute into a2188f252354e4ec3312a673da38529ad15da8cd on adamwathan:master.


Was versionCode decreased after v2.1.0?

I just was wondering why updates to the latest version failed. Looks like v2.2.1 has versionCode: 13 while v2.1.0 had versionCode: 1011, so Android considers it a downgrade. A closer look reveals: You no longer provide per-ABI builds but just a single "universal APK", hence that "Flutter rule" setting the versionCode to 1000 + versionCode for armeabi no longer applies.
May I ask to make per-ABI builds available again? That would also reduce the download size significantly (per-ABI was ~8MB, universal now is at ~24MB). That done, you could also pick a badge to link to your app in my repo next to the one from Google, as then recent versions would become available there again (currently they are removed as being "older") :wink:
Thanks in advance!

@IzzySoft Thanks for raising the issue, I definitely can do such an adjustment, however, I'm not sure about when the next release will be so you can actually benefit from it :-/. Let's see, thanks again!

Thanks for your fast reply, @tsinis! I'd say a day or a week or even a month would not matter much now. It just came up at my end as there were no updates for more than a year – at which point one of my scanners reports to me to take a look :wink:
And basically, no much change needed. You could even simply add the armeabi build for know (based on the same commit of that last tag – so no new release needed), and everything else can be done later.

Please give me a ping here when I shall take another look. Everything at your pace of course, no pressure – I just want to make sure in the end it's not me having missed your fix :wink:

Please give me a ping here when I shall take another look. Everything at your pace of course, no pressure – I just want to make sure in the end it's not me having missed your fix :wink:

Sure, I will, no worries.

@IzzySoft please take a look at release 2.2.2. Release CI is now using split-per-abi flag.

Looks good:
$ iod repo get is.tsin.colors_ai.colors_ai
is.tsin.colors_ai.colors_ai: looking for 'https://api.github.com/repos/tsinis/colors_ai/releases'
is.tsin.colors_ai.colors_ai: checking tag '2.2.2'
is.tsin.colors_ai.colors_ai: lastRelNo set to '2.2.2', checking for files
is.tsin.colors_ai.colors_ai: Upstream file date (2023-08-19 21:26) is newer than ours (2022-07-09 02:00).
is.tsin.colors_ai.colors_ai: returning ['2.2.2','https://github.com/tsinis/colors_ai/releases/download/2.2.2/colors-ai-armeabi-v7a.apk',1692473206]
is.tsin.colors_ai.colors_ai: 2.1.0/2.2.2, https://github.com/tsinis/colors_ai/releases: https://github.com/tsinis/colors_ai/releases/download/2.2.2/colors-ai-armeabi-v7a.apk
- Grabbing update for is.tsin.colors_ai.colors_ai: OK
- Checking 'repo/is.tsin.colors_ai.colors_ai_1014.apk' for libraries and malware …
is.tsin.colors_ai.colors_ai: check if repo contains FUNDING.yml
is.tsin.colors_ai.colors_ai: looking for 'https://api.github.com/repos/tsinis/colors_ai/contents/.github'
is.tsin.colors_ai.colors_ai: looking for 'https://api.github.com/repos/tsinis/colors_ai/contents/'
is.tsin.colors_ai.colors_ai: no FUNDING.yml detected.
is.tsin.colors_ai.colors_ai: no Fastlane configured, skipping Fastlane check.

And to answer unasked questions on the last 2 lines:

FUNDING.yml lets you maintain donation links
Fastlane Cheat Sheet lets you maintain description & graphics to present your app

So all fine, thanks a lot! Feel free to close this issue (as it's solved) or leave it open adding questions left on your end.

@IzzySoft Yes it's correct I don't have a FUNDING.yml in this repo and I'm not using Fastlane in any of my projects (GitHub workflows + Codemagic are enough for my needs). Thanks, I'm closing.

I'm not using Fastlane in any of my projects

If you had, my updater would fetch that along with new releases. Of course you can also ping me if changes are needed.

I'm not using Fastlane in any of my projects

If you had, my updater would fetch that along with new releases. Of course you can also ping me if changes are needed.

Sure I would change my mind - I will let you know. Thanks.


Add support for datastax-dse package to the diagnostic tool.

After testing this, @adyatlov and I realized that more changes are needed. The Datastax Cassandra node IDs follow a different pattern than Cassandra's, so CassandraBundle doesn't find them. I created https://jira.mesosphere.com/browse/DCOS-44747 to address it.


Allow for any translations.

Include TSV for NIV, NLT, etc.

I know that sometimes I, too, want to compare the Bible with corrupt translations (such as the ones you mentioned), but in such instances I usually just open up Bible Gateway.
I'm not going to implement this.


Update error handling

Fixes #668 and also makes some minor changes to how email verification is handled, adding a send_code route so that users can have the code resent to them.

Closing this PR - superceded by #670.


Incompete "Multiple access type modifier" Error Message

Running "Find & Replace" on a project file set had left me with one file with some instances of:
protected public static function xyz () {
...
}

Running phpcsfixer on the project resulted in a rather unhelpful "PHP Fatal error:  Multiple access type modifiers are not allowed in  on line 1444" error message.
An indication of which file, of about 3k files, that Line 1444 was located in would have been nice!

I edited the files outside phpcsfixer by finding and replacing a string. THIS is not the issue with phpcsfixer and was just background information.
Issue what when one has some thing like ...
protected public static function xyz () {
...
}

... hidden in one file among a few thousand files, phpcsfixer simply throws an error message saying "PHP Fatal error:  Multiple access type modifiers are not allowed in  on line 1234"
So now I know that there is an issue on Line xyz but line xyz of which file? That's the issue.
==============
TEMPLATE FILLED BELOW
The PHP version you are using ($ php -v):
=> PHP 7.1.1
PHP CS Fixer version you are using ($ php-cs-fixer -V):
=> PHP CS Fixer 2.12.1 Long Journey
The command you use to run PHP CS Fixer:
=> php php-cs-fixer fix --config /server/path/to/.php_cs.dist
The configuration file you are using, if any:
=> ....................................

If applicable, please provide minimum samples of PHP code (as plain text, not screenshots):

before running PHP CS Fixer (no changes):

protected public static function xyz () {
...
}

with unexpected changes applied when running PHP CS Fixer:

=> PHP Fatal error:  Multiple access type modifiers are not allowed in  on line 1234
// Note empty space between "in" and "on".
// File name should go there but missing

with the changes you expected instead:

=> PHP Fatal error:  Multiple access type modifiers are not allowed in **/path/to/file/abcd.ext** on line 1234

ha, now I got your request.
PHP CS Fixer is able (by purpose) to fix only valid PHP files, the message comes from linting process. I agree, having path to failing file would be nice indeed :+1:

I'm going to close this because issue mentions Fixer version which is not maintained anymore. Feel free to reproduce it on latest version (if problem still exists) and create new issue.


Improve Docker image

Improve Dockerfile

Added the RedHat base image that contains way less vulnerabilities than the official one

Official
RedHat

13H 30M 133L
8H 0M 0L

Get advantage of the layered cache: it doesn't make sense to install the requirements each time that the code is modified. The requirements should be cached as a layer and only rebuild it when the requirements file is modified. This will improve a lot the times between consequent Docker builds, since from second build and onward only the three last layers need to be rebuilt (bulding time of 1.1s on average with 10 tries) if just code is changed.
Use the .env.docker file by default in the image if an .env file is not present in the repository. I saw there was a problem in #43 due to the lack of .env file meanwhile building and running the Docker image. This should fix the problem even if I agree a better README.md should be in place.

Fix #51
There's a known bug on APScheduler 3.6.3 due to it's dependency on tzlocal 3.0b1: agronholm/apscheduler#461
Solved with APScheduler 3.7.0 with no further changelog than bugfixing (already checked)
https://github.com/agronholm/apscheduler/compare/v3.6.3...v3.7.0
Is this a patch, a minor version change, or a major version change
I would say that this could be considered a minor version change, but it's at the decision of the maintainers.
Is this related to an open issue?
#51
#42

Wonderful, thank you @dacamposol. @upkarlidder @albertum1 FYI.


create Dataverse card component

Dataverse card component
To do
Create dataverse card component that will be used for the Dataverse catalog page and the Explore block on the homepage
Acceptance criteria

[x] a functional component

[x] matches the design requirements

[x] can be used with different props on other pages

staging URL: https://app.staging.okp4.network/

Additional informations
None.

@lolottetheclash
I close this issue as it does not respect the criteria to specify a feature.
I let you create the feature based upon dedicated template.
Thx 🙏


🐛 [Bug] TensorRT engine exceptions are not raised

Bug Description
I would expect for torch to raise an exception when inference fails for any reason, such as wrong input tensor shape or wrong dtype. Instead, a warning is raised in console but the program continues successfully. This can have serious implications in production environments.
To Reproduce
I have a model compiled on float16 that accepts a static input shape of (1, 3, 538, 538)
import torch
import torch_tensorrt

model = torch.jit.load("model.ts")
input_tensor = torch.zeros((1, 3, 538, 538), dtype=torch.float16, device="cuda")
output_tensor = model(input_tensor)

This is what happens if I pass a wrong shape
>>> out = model(torch.zeros((1, 3, 500, 500), dtype=torch.float16, device="cuda"))
ERROR: [Torch-TensorRT] - 3: [executionContext.cpp::setInputShape::2020] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::setInputShape::2020, condition: engineDims.d[i] == dims.d[i]. Static dimension mismatch while setting input shape.
)
>>> # NO EXCEPTION IS RAISED

This is what happens if I pass a wrong dtype
>>> out = model(torch.zeros((1, 3, 538, 538), dtype=torch.float32, device="cuda"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/wizard/mambaforge/envs/remini/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/<model>.py", line 8, in forward
    input_0: Tensor) -> Tensor:
    __torch___<model>_trt_engine_ = self_1.__torch___<model>_trt_engine_
    _0 = ops.tensorrt.execute_engine([input_0], __torch___<model>_trt_engine_)
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    _1, = _0
    return _1

Traceback of TorchScript, original code (most recent call last):
RuntimeError: [Error thrown at core/runtime/execute_engine.cpp:136] Expected inputs[i].dtype() == expected_type to be true but got false
Expected input tensors to have type Half, found type float

>>> # NO EXCEPTION IS RAISED

Expected behavior
An exception should be raised if the TensorRT Engine returns an error.
Environment

Build information about Torch-TensorRT can be found by turning on debug messages

Torch-TensorRT Version (e.g. 1.0.0): 1.4.0
PyTorch Version (e.g. 1.0): 2.0.1
CPU Architecture: x86_64
OS (e.g., Linux): LInux
How you installed PyTorch (conda, pip, libtorch, source): pip (custom whl)
Build command you used (if compiling from source): -
Are you using local sources or building from archives: -
Python version: 3.9
CUDA version: 11.8
GPU models and configuration: Tesla T4 / Tesla L4
Any other relevant information: tensorrt <IP_ADDRESS>

Here's a comment to keep the issue alive.


QUESTION: Message with payload but no meta-data. . .

One of the value of the Message is to create a structure that can encapsulate data and the supplemental Information about such data.
Looking at the current Message I only see a payload. Is there a reason for omitting it?
public interface Message<T> {

  /**
   * The payload for this message.
   */
  T getPayload();

  /**
   * Supplemental Information about this message
   */
  Map<String, Object> getHeaders();
}

The only reason it's not there is because in the initial PR, we wanted the API to be very small so that individual features can be discussed over individual pull requests, not because we don't think it should be there.
One thing that I'm not sure about, we have the event API being proposed in #14 based on https://cloudevents.io, is it enough to simply have the metadata on Event (which is a subclass of Message), or does it need to be on Message? One nice thing about putting it on Event is that we can be more opinionated about the format, we can for example use Map<String, CloudEventObject> where CloudEventObject has three subclasses according to the cloud events spec for object - string, byte array and map. This is much nicer than just Object, because with just Object you could get anything. That's not to say that instances of Message couldn't provide metadata, different implementations can supply their own meta data maps using stronger typing if it makes sense, for example, implementations that only support string values can expose Map<String, String>, which would be nicer to use than Map<String, Object>.

@jroper Thanks for following up.
As the current javadoc states A message envelope and as such envelope usually contains meta-information as well as the payload and that is the true value of Message as a simple structure to communicate information in a portable way. Simply having this structure as a wrapper over a single T doesn't seem to add much value, Why not just sent T? In any event, I do believe we are on the same page with regard to intentions.
With regard to the type of value of these properties, sure it could be another H, although in my experience these property values can be of different types in a single message.

Slightly unrelated. . . I see that current state of the Message has few factory operations (i.e., Message.of(..)).  Is it necessary or could they be moved to some utility class?
Personally I was always envisioning a simple structure that anyone can agree on. So having Message strategy which only has payload and headers and could also be easily described via simple json (making it essentially portable in the polyglot world) would be a far better approach.
I do understand that these factory operations would not affect any of what I am describing above, but it makes the class itself a bit "too busy" (for the lack of a better term).

The reason the message class currently exists is for correlation so that the right message can be acknowledged. If I am receiving messages, applying some transformation, and sending to a new destination, I need to ensure that the messages received are not acknowledged (committed, etc) until the transformed message has reached the destination, otherwise messages could get dropped. Now, if it's a simple 1:1 transformation, that's straight forward, the messages can be correlated easily. But what if the transformation is not 1:1? What if I filter the messages? This is what the Message envelope does currently, it allows the ack function to be carried from input messages to output messages so that the right message gets acknowledged at the right time.

Fair enough. But then would you consider separating the Envelope and the Message? Just doesn't seem right that the Message caries this responsibility. Think of a mail-man/post-office anolongy where they will attempt to deliver and they will take care of ack or nack. The Message itself just represents the data.

Are you suggesting the current Message gets renamed to Envelope?

All I am saying is that IMHO there is a data (Message) and the courier (???) of such data. The _courier is responsible for all delivery controls (ack/nack), transfers (to possibly other curriers) etc, while the Message remains immutable once created.

I think that's worth considering. My question is, while they may be different concepts, what practical value will be gained by separating them? Are there use cases where the separation in the API would enable things that wouldn't be possible if they weren't separated? Can it lead to a more fluent API? etc...
By the way, what are your thoughts on handling acks, particularly when transforming an incoming stream to an outgoing stream? What does Spring Cloud Stream do here, in particular, when Reactor is used, and say a filter is done, so there is no 1:1 correlation of input to output events? Does it use a similar wrapper (envelope) to carry the correlation? Or something else?

I think that's worth considering. My question is, while they may be different concepts, what practical value will be gained by separating them? Are there use cases where the separation in the API would enable things that wouldn't be possible if they weren't separated? Can it lead to a more fluent API? etc...
By the way, what are your thoughts on handling acks, particularly when transforming an incoming stream to an outgoing stream? What does Spring Cloud Stream do here, in particular, when Reactor is used, and say a filter is done, so there is no 1:1 correlation of input to output events? Does it use a similar wrapper (envelope) to carry the correlation? Or something else?

I think portability and communication is the main case. A simple Tuple2-like structure for a Message existed for decades in many messaging frameworks/products with some slight variations. Standardizing around T, Map<String, Object> would ensure that we all have something in common. It also has implications outside of Java since the above structure can be easily represented in JSON thus supporting polyglot distributed messaging applications.
So, yes, Envelope or any other name that would encapsulate Message would suffice. And then you can support Envelope.ack/nack and may other operators. It would also follow the logic that "something" (not the Message itself) is responsible for ops-type/lifecycle operation pertaining to a Message.
With regard to Spring Cloud Stream, we are currently working with Project Reactor with  Flux<Message<?>> being the center point.
The ops-type/lifecycle operation are currently tied to doOnComplete/Cancel(..), retryBackoff(..) etc., providing relevant information (i.e., delivery tag, offset etc) is contained in message headers.
I would certainly be open to revisiting that with regard to Envelope, as long as Message itself remains simply a data object.

FWIW, this is the Message I am speaking about.

After having played a little bit with the spec (and my implementation), I believe the Message should also have headers as a Map<String, Object> (might be also serializable as in the code shared by @olegz).
Adding headers also add a bit of complexity in term of propagation as the header might need to be propagated. This is particularly true for methods consuming the payload directly such as:
@Incoming(...)
@Outgoing(...)
public Item process(SomeData data) {
   // some clever code creating an Item out of some data
}

In this code, the headers associated with the Message conveying the SomeData may need to be propagated to the Message conveying the resulting Item.

Header propagation is indeed something we had to struggle with in the past, but got it down pretty good within the last decade, so we can certainly share some pros and cons. Perhaps a separate thread would be nice for it once all agreed on the Message structure with headers.

Migrated to https://github.com/eclipse/microprofile-reactive-messaging/issues/5.


Kafka topic docs clarity

Hi
For the Kafka Topic documentation, could you please specify the units for both time and size.

Also, how do I set the retention and message size policies to infinite? I suggest these be documented in the page too.

For the sizes, unit is byte and for time, unit is millisecond-ms. For the infinite values, you can give -1 as the value. However, if it is not permitted -regarding your payment plan-, you will have created with the max value available to you - 1 TB for instance- .
Thank you for bringing this to our attention! We will add that info soon.

We have added the unit info to our developer api. Closing for now.


[Bug]: CodeActAgent unaware of the tools

Is there an existing issue for the same bug?

[X] I have checked the troubleshooting document at https://docs.all-hands.dev/modules/usage/troubleshooting
[X] I have checked the existing issues.

Describe the bug
I'm trying to do anything useful with the agent. I tried with base models like :

llama3.1-70
phi3-medium
gemma-27

The initial prompt is some simple one like:

Create a fresh NextJS Project.
You have access to terminal, workspace where you should create a project files. Also browser and Jupyter iPython to plan, execute, process and gather data.

 I apologize for the confusion. I am still under development and learning how to best fulfill request
Let me try again. Since I can't directly interact with your system, I will provide you with a detailed set of instructions that, if followed carefully, should result in the creation of the project files as you requested.
Please follow these steps precisely:

The Agent is unaware of the tools that he can use. Few times i was able to force him by direct instructions.
Current OpenDevin version
0.8.x

Installation and Configuration
just default docker execution

Model and Agent
No response
Operating System
No response
Reproduction Steps
No response
Logs, Errors, Screenshots, and Additional Context
No response

Hey @imkebe , have you also tried to do the same thing with either gpt-4o or claude-3-5-sonnet? I'm wondering if it's an issue of the LM?

Hey @imkebe , have you also tried to do the same thing with either gpt-4o or claude-3-5-sonnet? I'm wondering if it's an issue of the LM?

gpt-4o is working fine.
I've simulated the completions with OpenWebUI.
Instructions are rather simple and I thought mistral-nemo, llama3.1-x, phi3 should handle such simple instructions. It turns out the either the prompt needs to be modified for open-source LLM's or they are just dumb.
I've attached Langfuse traces with full prompts and responses:
llama3.1_70b-instruct-q6_K.pdf
phi3-14b-medium-128k-instruct-q6_K.pdf
mistral-nemo_12b-instruct-2407-fp16.pdf

Awesome, thanks. We're still working on trying to make competitive open-source LMs, but it's not an easy task. We have a larger tracking issue for that here: #1085
In the mean time, I'll close this since we have the other issue for that discussion.


Deleting eks anywhere cluster failure with getting total count of nodes

What happened:
Attempting to delete an eks anywhere clustering using the following command
eksctl anywhere delete cluster mgmt-dev --bundles-override bundle-release.yaml results in a failure.
Error: failed to delete cluster: getting the total count of nodes: getting KubeadmControlPlane for cluster mgmt-dev: getting kubeadmcontrolplane: Error from server (NotFound): kubeadmcontrolplanes.controlplane.cluster.x-k8s.io "mgmt-dev" not found
What you expected to happen:
I expect to have the eks cluster deleted.
How to reproduce it (as minimally and precisely as possible):
Not sure how I got to this state.
Anything else we need to know?:
The cluster was originally created using the following command per airgap documentation,
eksctl anywhere create cluster -f eksa-mgmt-cluster.yaml --bundles-override bundle-release.yaml
Environment:

EKS Anywhere Release: v0.13.1
EKS Distro Release: N/A

Was your kubeconfig context pointing to your management cluster?
Maybe run k get kubeadmcontrolplanes.controlplane.cluster.x-k8s.io -n eksa-system

@miryalasaigoutham Can you repost the commands you're running and the files on disk you're using? This command is known to work so the working theory is you're not using the correct files.