[Netarchivesuite-users] Problems while starting NAS after server update

Mikis Seth Sørensen mss at statsbiblioteket.dk
Fri Oct 3 13:01:17 CEST 2014


Sounds good :-)

We have struggled a bit with the security policy ourselves as part of
upgrading the java version to Java8 in the coming NAS 5. The Java8
security model is more restrict (started in java7u51), which broke the
serverside apps, and as you have experience it can be quite a overwelming
task figuring out what is going on when you encounter java security policy
problems. So the specific definition of the security policy has been
remove for all applications except the BitArchive apps, where it is needed
to control the permissions for injected BatchJob code.

Best
Mikis

On 10/3/14, 12:37 PM, "Meelis Mihhailov" <meelis at nlib.ee> wrote:

>Hi Mikis
>
>There was indeed a typo and noticed it right after sending the first
>e-mail. Changing it however did not fix the problems :(
>
>Today NAS is working again BUT it still shows DB connecting errors while
>running startall script. However NAS is working, data is accessible and
>I can edit it. So DB actually IS accessible and NAS actually IS
>connected. Strange...
>
>I actually do not understand what was the fix. I first tried to add
>permissions to security files located in the deploy dir and installation
>dir. This did not solve my problem. After that I actually added new
>policy file to default java location and defined specially what
>permissions to what domain will be used. Still ... nothing. So I added
>special command line parameters to
>start_external_admin/harvest_database.sh files (feeding the special
>policy file) and this fixed the deploy.log errors. However it still did
>not fix the connection error and NAS was not accessible.
>
>So after I was quite frustrated I decided to start all over again. Undo
>all the changes made EXCEPT default java.policy file. (Had already fixed
>the typo before ... just to point out that it was not typo that fixed it
>:) ). So restarted NAS, saw all the old "Can't connect to DB" errors
>and just to frustrate myself more ... accessed NAS web interface and saw
>that it was up and running!!
>
>What fixed the problem that NAS was not working? No idea :)
>
>I'm aware that there is a new version out :) We have the update
>scheduled but use this old version at the moment for testing and
>currently exporting harvest information in order to import it to new
>installation. Also I'm quite exited to see what happens to version 5  :)
>
>-----------------------------------------------------
>Meelis Mihhailov
>Eesti Rahvusraamatukogu / National Library Of Estonia
>
>Telefon: 630 7178 / Phone: +372 630 7178
>E-post: meelis at nlib.ee / E-mail: meelis at nlib.ee
>
>Tõnismägi 2, 15189 Tallinn, ESTONIA
>
>www.eestirahvusraamatukogu.ee
>-----------------------------------------------------
>
>On 3.10.2014 13:11, Mikis Seth Sørensen wrote:
>> Hi Meelis
>>
>> Thanks for the detailed breakdown of your problem :-).
>>
>> I¹ve noticed 2 things:
>> * You write that you have added a 'permission java.net.SocletPermission¹
>> line to your java.policy file. That should be a
>> 'java.net.SocketPermission¹, not SocletPermission.
>> * Yoy are running a really old version of NetarchiveSuite. The latest
>> version is 4.4 which can be downloaded here:
>> https://sbforge.org/display/NAS/NetarchiveSuite+4.4+Release+Notes.
>>
>> Does this help?
>>
>> Mikis Seth Sørensen
>> NetarchiveSuite Developer
>> Victor Albecks Vej 1
>> 8000 Aarhus C
>>
>> Denmark
>>
>> On 10/1/14, 12:30 PM, "Meelis Mihhailov" <meelis at nlib.ee> wrote:
>>
>>> Hi,
>>>
>>> I have a problem starting NAS after server update (and restart) and
>>>hope
>>> someone can help me to resolve this issue :)
>>>
>>> Server had java update as well and now NAS cannot start because of
>>> errors while connecting to derby database.
>>>
>>> Using NAS 3.21 as a quickstart setup
>>> Java version "1.6.0_31"
>>> Server : Debian GNU/Linux 7
>>>
>>> While using the "startall" script I get the following errors:
>>>
>>> 
>>>------------------------------------------------------------------------
>>>--
>>> --------------------------------
>>>
>>> Beginning database upgrade
>>> log4j:WARN No appenders could be found for logger
>>> (com.mchange.v2.log.MLog).
>>> log4j:WARN Please initialize the log4j system properly.
>>> Exception in thread "main" dk.netarkivet.common.exceptions.IOFailure:
>>> Can't connect to database with DBurl:
>>> 'jdbc:derby://localhost:8121/harvestDatabase/fullhddb' using driver
>>> 'org.apache.derby.jdbc.ClientDriver'
>>> SQLException trace:
>>> SQL State:null
>>> Error Code:0
>>> java.sql.SQLException: Connections could not be acquired from the
>>> underlying database!
>>>          at 
>>>com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>>>          at
>>> 
>>>com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnecti
>>>on
>>> (C3P0PooledConnectionPool.java:529)
>>>          at
>>> 
>>>com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(Abst
>>>ra
>>> ctPoolBackedDataSource.java:128)
>>>          at
>>> 
>>>dk.netarkivet.harvester.datamodel.HarvestDBConnection.get(HarvestDBConne
>>>ct
>>> ion.java:110)
>>>          at
>>> 
>>>dk.netarkivet.harvester.datamodel.DBSpecifics.updateTable(DBSpecifics.ja
>>>va
>>> :127)
>>>          at
>>> 
>>>dk.netarkivet.harvester.datamodel.DBSpecifics.updateTables(DBSpecifics.j
>>>av
>>> a:625)
>>>          at
>>> 
>>>dk.netarkivet.harvester.tools.HarvestdatabaseUpdateApplication.main(Harv
>>>es
>>> tdatabaseUpdateApplication.java:45)
>>> Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException:
>>>A
>>> ResourcePool could not acquire a resource from its primary factory or
>>> source.
>>>          at
>>> 
>>>com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResour
>>>ce
>>> Pool.java:1319)
>>>          at
>>> 
>>>com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(Bas
>>>ic
>>> ResourcePool.java:557)
>>>          at
>>> 
>>>com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicReso
>>>ur
>>> cePool.java:477)
>>>          at
>>> 
>>>com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnecti
>>>on
>>> (C3P0PooledConnectionPool.java:525)
>>>          ... 5 more
>>> End of SQLException trace
>>>          at
>>> 
>>>dk.netarkivet.harvester.datamodel.HarvestDBConnection.get(HarvestDBConne
>>>ct
>>> ion.java:119)
>>>          at
>>> 
>>>dk.netarkivet.harvester.datamodel.DBSpecifics.updateTable(DBSpecifics.ja
>>>va
>>> :127)
>>>          at
>>> 
>>>dk.netarkivet.harvester.datamodel.DBSpecifics.updateTables(DBSpecifics.j
>>>av
>>> a:625)
>>>          at
>>> 
>>>dk.netarkivet.harvester.tools.HarvestdatabaseUpdateApplication.main(Harv
>>>es
>>> tdatabaseUpdateApplication.java:45)
>>> Caused by: java.sql.SQLException: Connections could not be acquired
>>>from
>>> the underlying database!
>>>          at 
>>>com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>>>          at
>>> 
>>>com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnecti
>>>on
>>> (C3P0PooledConnectionPool.java:529)
>>>          at
>>> 
>>>com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(Abst
>>>ra
>>> ctPoolBackedDataSource.java:128)
>>>          at
>>> 
>>>dk.netarkivet.harvester.datamodel.HarvestDBConnection.get(HarvestDBConne
>>>ct
>>> ion.java:110)
>>>          ... 3 more
>>> Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException:
>>>A
>>> ResourcePool could not acquire a resource from its primary factory or
>>> source.
>>>          at
>>> 
>>>com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResour
>>>ce
>>> Pool.java:1319)
>>>          at
>>> 
>>>com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(Bas
>>>ic
>>> ResourcePool.java:557)
>>>          at
>>> 
>>>com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicReso
>>>ur
>>> cePool.java:477)
>>>          at
>>> 
>>>com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnecti
>>>on
>>> (C3P0PooledConnectionPool.java:525)
>>>          ... 5 more
>>> 
>>>------------------------------------------------------------------------
>>>--
>>> --------------------------------
>>>
>>> and etc.
>>>
>>> After doing some investigation I found in derby.log:
>>>
>>>
>>> 
>>>------------------------------------------------------------------------
>>>--
>>> --------------------------------
>>> Wed Oct 01 13:09:23 EEST 2014 : access denied
>>>(java.net.SocketPermission
>>> localhost:8121 listen,resolve)
>>> java.security.AccessControlException: access denied
>>> (java.net.SocketPermission localhost:8121 listen,resolve)
>>>          at
>>> 
>>>java.security.AccessControlContext.checkPermission(AccessControlContext.
>>>ja
>>> va:399)
>>>          at
>>> 
>>>java.security.AccessController.checkPermission(AccessController.java:557
>>>)
>>>          at
>>> java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
>>>          at
>>> java.lang.SecurityManager.checkListen(SecurityManager.java:1134)
>>>          at java.net.ServerSocket.bind(ServerSocket.java:335)
>>>          at java.net.ServerSocket.<init>(ServerSocket.java:202)
>>>          at
>>> 
>>>javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFact
>>>or
>>> y.java:189)
>>>          at
>>> 
>>>org.apache.derby.impl.drda.NetworkServerControlImpl.createServerSocket(U
>>>nk
>>> nown
>>> Source)
>>>          at
>>> org.apache.derby.impl.drda.NetworkServerControlImpl.access$000(Unknown
>>> Source)
>>>          at
>>> org.apache.derby.impl.drda.NetworkServerControlImpl$1.run(Unknown
>>>Source)
>>>          at java.security.AccessController.doPrivileged(Native Method)
>>>          at
>>> 
>>>org.apache.derby.impl.drda.NetworkServerControlImpl.blockingStart(Unknow
>>>n
>>> Source)
>>>          at
>>> org.apache.derby.impl.drda.NetworkServerControlImpl.executeWork(Unknown
>>> Source)
>>>          at org.apache.derby.drda.NetworkServerControl.main(Unknown
>>> Source)
>>> tail: /arhiiv/CRAWLER/NLIB/derby.log: file truncated
>>> Wed Oct 01 13:09:28 EEST 2014 : access denied
>>>(java.net.SocketPermission
>>> localhost:8120 listen,resolve)
>>> java.security.AccessControlException: access denied
>>> (java.net.SocketPermission localhost:8120 listen,resolve)
>>>          at
>>> 
>>>java.security.AccessControlContext.checkPermission(AccessControlContext.
>>>ja
>>> va:399)
>>>          at
>>> 
>>>java.security.AccessController.checkPermission(AccessController.java:557
>>>)
>>>          at
>>> java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
>>>          at
>>> java.lang.SecurityManager.checkListen(SecurityManager.java:1134)
>>>          at java.net.ServerSocket.bind(ServerSocket.java:335)
>>>          at java.net.ServerSocket.<init>(ServerSocket.java:202)
>>>          at
>>> 
>>>javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFact
>>>or
>>> y.java:189)
>>>          at
>>> 
>>>org.apache.derby.impl.drda.NetworkServerControlImpl.createServerSocket(U
>>>nk
>>> nown
>>> Source)
>>>          at
>>> org.apache.derby.impl.drda.NetworkServerControlImpl.access$000(Unknown
>>> Source)
>>>          at
>>> org.apache.derby.impl.drda.NetworkServerControlImpl$1.run(Unknown
>>>Source)
>>>          at java.security.AccessController.doPrivileged(Native Method)
>>>          at
>>> 
>>>org.apache.derby.impl.drda.NetworkServerControlImpl.blockingStart(Unknow
>>>n
>>> Source)
>>>          at
>>> org.apache.derby.impl.drda.NetworkServerControlImpl.executeWork(Unknown
>>> Source)
>>>          at org.apache.derby.drda.NetworkServerControl.main(Unknown
>>> Source)
>>> 
>>>------------------------------------------------------------------------
>>>--
>>> --------------------------------
>>>
>>> So this seems to point to some sort of java security problem. Right ...
>>> looked at the default java.policy file and added these lines:
>>>
>>> permission java.net.SocletPermission "localhost:8120",
>>>"listen,resolve";
>>> permission java.net.SocletPermission "localhost:8121",
>>>"listen,resolve";
>>>
>>> Nothing ... still errors. So I looked around the installation to see if
>>> there are any other files I can alter. Found two additional files:
>>>
>>> /INSTALL_DIR/DEPLOY_DIR/conf/security.policy
>>> /INSTALL_DIR/localhost/security.policy
>>>
>>> did the changes as in default java.policy file and again ... errors
>>> while starting NAS.
>>>
>>> Even added firewall rules just to be safe but nothing. Cannot start NAS
>>> and do not understand where can I change the permissions in order to
>>> give derby db access for NAS.
>>>
>>> Any ideas and suggestions are welcome :)
>>>
>>>
>>>
>>> -- 
>>> -----------------------------------------------------
>>> Meelis Mihhailov
>>> Eesti Rahvusraamatukogu / National Library Of Estonia
>>>
>>> Telefon: 630 7178 / Phone: +372 630 7178
>>> E-post: meelis at nlib.ee / E-mail: meelis at nlib.ee
>>>
>>> Tõnismägi 2, 15189 Tallinn, ESTONIA
>>>
>>> www.eestirahvusraamatukogu.ee
>>> -----------------------------------------------------
>>> _______________________________________________
>>> NetarchiveSuite-users mailing list
>>> NetarchiveSuite-users at ml.sbforge.org
>>> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>>
>> _______________________________________________
>> NetarchiveSuite-users mailing list
>> NetarchiveSuite-users at ml.sbforge.org
>> http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users
>
>_______________________________________________
>NetarchiveSuite-users mailing list
>NetarchiveSuite-users at ml.sbforge.org
>http://ml.sbforge.org/mailman/listinfo/netarchivesuite-users




More information about the NetarchiveSuite-users mailing list