WritePendingException when connecting two receivers to the testbed

[dragos]

[ERROR] [06/17/2015 17:34:57.709] [application-akka.actor.default-dispatcher-15] [akka://application/user/serverManager/$c] null
java.nio.channels.WritePendingException
    at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:351)
    at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:386)
    at com.typesafe.spark.testbed.ConnectionManagerActor$$anonfun$receive$1.applyOrElse(DataGeneratorActor.scala:180)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
    at com.typesafe.spark.testbed.ConnectionManagerActor.aroundReceive(DataGeneratorActor.scala:170)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
    at akka.actor.ActorCell.invoke(ActorCell.scala:487)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
    at akka.dispatch.Mailbox.run(Mailbox.scala:220)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[dragos]

Hmm, it seems to be unrelated, the exception happens with only one stream as well. I restarted the testbed and then it worked with two receivers as well...

[huitseeker]

Could you re-test with adding --conf spark.executor.memory=8G (or something massive) on the recipient side ?
Also, could you provide the contents of your testplan ?

[dragos]

The exception happens on the testbed application, not the Spark client.

[huitseeker]

Yes. Nonetheless, having a spark client with a massive amount of memory means congestion happens later, thus TCP buffer overflow happens later, thus write errors are delayed.