Port 4040 already in use

Technical Discussions Apache Spark
#1

While Running Spark Application i got this error
18/05/31 14:37:26 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use

and then in last

18/05/31 15:04:15 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@1deb2c43: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:437)
at sun.nio.ch.Net.bind(Net.java:429)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:237)
at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)

How to resolve this

#2

Hi @anurag_pandey,

This is just a warning as port 4040 is already in use by other program. Spark-shell will get launched in the next available port since port 4040 is already in use.

Hope this helps.

Thanks

#3

spark-submit \

–class “Simple”
–master local[4]
target/scala-2.11/simple_2.11-1.0.jar
18/05/31 16:51:02 WARN SparkConf: The configuration key ‘spark.yarn.applicationMaster.waitTries’ has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key ‘spark.yarn.am.waitTime’ instead.
18/05/31 16:51:02 INFO SparkContext: Running Spark version 1.5.2
18/05/31 16:51:02 WARN SparkConf: The configuration key ‘spark.yarn.applicationMaster.waitTries’ has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key ‘spark.yarn.am.waitTime’ instead.
18/05/31 16:51:02 INFO SecurityManager: Changing view acls to: anurag99868515511048
18/05/31 16:51:02 INFO SecurityManager: Changing modify acls to: anurag99868515511048
18/05/31 16:51:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(anurag99868515511048); users with modify permissions: Set(anurag99868515511048)
18/05/31 16:51:03 INFO Slf4jLogger: Slf4jLogger started
18/05/31 16:51:03 INFO Remoting: Starting remoting
18/05/31 16:51:03 INFO Utils: Successfully started service ‘sparkDriver’ on port 33452.
18/05/31 16:51:03 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@172.31.38.183:33452]
18/05/31 16:51:03 INFO SparkEnv: Registering MapOutputTracker
18/05/31 16:51:03 WARN SparkConf: The configuration key ‘spark.yarn.applicationMaster.waitTries’ has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key ‘spark.yarn.am.waitTime’ instead.
18/05/31 16:51:03 WARN SparkConf: The configuration key ‘spark.yarn.applicationMaster.waitTries’ has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key ‘spark.yarn.am.waitTime’ instead.
18/05/31 16:51:03 INFO SparkEnv: Registering BlockManagerMaster
18/05/31 16:51:03 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-55acb102-6f47-4374-9377-bb84e8115fb5
18/05/31 16:51:03 INFO MemoryStore: MemoryStore started with capacity 265.1 MB
18/05/31 16:51:03 WARN SparkConf: The configuration key ‘spark.yarn.applicationMaster.waitTries’ has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key ‘spark.yarn.am.waitTime’ instead.
18/05/31 16:51:03 INFO HttpFileServer: HTTP File server directory is /tmp/spark-62098cab-fe73-4bfc-afed-14cb5b110a4e/httpd-831f0753-c0cf-4613-b5c3-b2695bef189a
18/05/31 16:51:03 INFO HttpServer: Starting HTTP Server
18/05/31 16:51:03 INFO Server: jetty-8.y.z-SNAPSHOT
18/05/31 16:51:03 INFO AbstractConnector: Started SocketConnector@0.0.0.0:55589
18/05/31 16:51:03 INFO Utils: Successfully started service ‘HTTP file server’ on port 55589.
18/05/31 16:51:03 INFO SparkEnv: Registering OutputCommitCoordinator
18/05/31 16:51:03 INFO Server: jetty-8.y.z-SNAPSHOT
18/05/31 16:51:04 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:437)
at sun.nio.ch.Net.bind(Net.java:429)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:237)
at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:247)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.(SparkContext.scala:474)
at Simple$.main(Simple.scala:9)
at Simple.main(Simple.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:685)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/05/31 16:51:04 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@443dbe42: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:437)
at sun.nio.ch.Net.bind(Net.java:429)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:237)
at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:247)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.(SparkContext.scala:474)
at Simple$.main(Simple.scala:9)
at Simple.main(Simple.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:685)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/05/31 16:51:04 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
18/05/31 16:51:04 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
18/05/31 16:51:04 INFO Con