extract-to-scalar output hdf files out of memory error

[wangyuhan01]

We ran into the following errors while running "extract-to-scalar" and convert n5 files to hdf5 format, but output n5 format seem to work fine. Thanks for looking into it!

19/08/08 16:49:39 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 1,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at ch.systemsx.cisd.hdf5.HDF5BaseWriter.setupSyncThread(HDF5BaseWriter.java:170)
at ch.systemsx.cisd.hdf5.HDF5BaseWriter.(HDF5BaseWriter.java:165)
at ch.systemsx.cisd.hdf5.HDF5WriterConfigurator.writer(HDF5WriterConfigurator.java:133)
at ch.systemsx.cisd.hdf5.HDF5FactoryProvider$HDF5Factory.open(HDF5FactoryProvider.java:48)
at ch.systemsx.cisd.hdf5.HDF5Factory.open(HDF5Factory.java:47)
at org.janelia.saalfeldlab.n5.hdf5.N5HDF5Writer.(N5HDF5Writer.java:92)
at org.janelia.saalfeldlab.label.spark.N5Helpers.n5Writer(N5Helpers.java:38)
at org.janelia.saalfeldlab.conversion.ExtractHighestResolutionLabelDataset$Args.lambda$call$49828ecb$1(ExtractHighestResolutionLabelDataset.java:101)
at org.janelia.saalfeldlab.conversion.ExtractHighestResolutionLabelDataset.lambda$extract$a90bce4d$1(ExtractHighestResolutionLabelDataset.java:248)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:921)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:921)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[hanslovsky]

Thanks for raising this @wangyuhan01, I will look into this.

[hanslovsky]

I will close here in favor of saalfeldlab/paintera-conversion-helper#28 because the code lives in that repository.