[Netarchivesuite-devel] Plans for NetarchiveSuite after the 4.4 release

[tra-spam]

Hi.

As Mikis mentioned, I am currently having a look at the subversion->git and ant->maven migrations.

What may not be immediately evident is why we want to do this.  The ant->maven migration has at least these major benefits:


·         Maven does dependency management.  This has many benefits but the major benefit as I see it is that it allows us to be able to split the build process up in multiple steps potentially separated in time and space while still having meta-data available from earlier in the process. Additionally the jars needed by the project do not need to be in the source code repository any more.

·         A single configuration supported by all major IDE’s and a commandline build, instead of having an Eclipse configuration, an IntelliJ configuration and an ant configuration (all manually maintained) exist independently next to each other.

·         Jenkins understand Maven projects natively, and can trigger the job building a component dependent on a maven artifact just built by another job.  This is an important step in order to be able orchestrate our Continuous Deployment effort with e.g. Jenkins.

Maven is not as lenient as ant, so we need to move the existing files around to be able to conform to the maven philosophy of “one project; one maven artifact”.  We would like to keep the history of the project instead of just copying the sources over, which is not trivial to do with subversion, but is a supported use case in git.  Hence this situation again prompts migrating from subversion to git before reorganizing the files (with history) for Maven.

An initial observation is that the repository contains a lot of binary artifacts resulting in it having a size of 233 MB.   This may or may not be an issue in terms of cloning speed from github.  If it is, we will create a new, trimmed repository where only the source files available today is kept (along with their history), and all binary artifacts not used today are removed.

So, to summarize – the steps after the 4.4 release are:

·         Change the version control system from subversion to git.  The existing subversion repository will be kept for archival purposes.  The git repository represent the exact same filesystem as the subversion repository.

·         Work in the new git repository for an yet undetermined amount of time.

·         Optionally create a trimmed repository.  If so, the untrimmed git repository is kept for archival purposes.

·         Reorganize the existing sources and tests into a Maven multi-module build along with their history.  Failing tests are initially disabled.  Dependencies needed but not present in Maven Central are dealt with on a one by one basis.

·         Failing tests are fixed or discarded one by one.

/Thorbjørn

From: Netarchivesuite-devel [mailto:netarchivesuite-devel-bounces at ml.sbforge.org] On Behalf Of Mikis Seth Sørensen
Sent: Thursday, May 01, 2014 11:41 AM
To: netarchivesuite-devel at ml.sbforge.org
Subject: [Netarchivesuite-devel] Plans for NetarchiveSuite after the 4.4 release

Hi

Are a short summary for our current NetarchiveSUite development  plans after we wrap up the 4.4  release. Discussion this detail would be a natural item for our NetarchiveSuite friday talk in Paris.

As I mentioned on the last meeting, we have plan to use a couple of months to update and improve the NetarchiveSuite codesetup, so we hopefully afterwards can use more of our time producing new feaures, instead of trying to get the NetarchiveSuite system to work as expected. Currently we're spend a lot of time trying to fix stuff we break during development, because the current NAS codebase is somewhat difficult to maintain. Nicolas identified a lot of opertunities for improvement before he left, and we have also build up a list of things we would like to improve on.

So the current things we would like to work on for the next month or two are:

 *   Switch from the current Subversion repository on SBForge to git host on Github. See NAS-2041<https://sbforge.org/jira/browse/NAS-2041> Switch to git as the NetarchiveSuite SCM.
 *   Switch from using Ant to build NAS to using Maven. See NAS-8<https://sbforge.org/jira/browse/NAS-8> Switch to Maven as build tool,
 *   Reduce the complexity of the current DB maintenance setup. This includes a number of more concrete tasks:
    *   Decide whether we are continuing with our current DB specific SQL or switch to a using a DB abstraction layer like Hibernate
    *   Improve our DB upgrade mechanisme. See NAS-2265<https://sbforge.org/jira/browse/NAS-2265> DAO refactoring - Drop programmatic DB updates and versioning table.
    *   Replace the current Derby test with a test DB which doesn't come with it's own SQL flavour. See NAS-2268<https://sbforge.org/jira/browse/NAS-2268>  Switch to H2 as TestDB.
    *   Reduce the current host of DB dumps and DB creation scripts used in testing to a single DB schema definition + some DB enrichment/migration scripts.
    *   Perhaps drop MySQL support, if ONB switch to PostgreSQL. This would bring the number of SQL flavours we need to supprt to 1.

 *   Extend the number of automatic system test, to enable us to cheap and continuously run tests on deployed NetarchiveSuite systems. See
 *   NAS-1859 Create automatic and continuous sanity test of the NAS system<https://sbforge.org/jira/browse/NAS-1859>. This can hopefully be made test system independent so the tests may be run on the environments used at ONB and Bnf. The work has already started on this as I would like to continuously deploy and test the Quickstart system, besides the current DK testenvironment tests.
 *   NAS-2276<https://sbforge.org/jira/browse/NAS-2276> Switch to more modern unit test framework<https://sbforge.org/jira/browse/NAS-2276>.
 *   NAS-2124<https://sbforge.org/jira/browse/NAS-2124> Switch to slf4j as logging framework.
Thorbjørn is already deeply involved in preparing for the Git and Maven migration, and we plan to perform this switch shortly after the 4.4 codefreeze is lifted (next week perhaps?). Have a look at the two issues related to these migrations for details.

Any thoughts on these ideas or other pressing stuff we should look at? Would BnF or ONB be interested in participation in any of these improvement efforts?


After this we plan to start working towards supporting H3. We'll hopefully have both Nicholas and Thorbjørn available full time to work on this. But we would of course also very much like to have BnF and ONB involved is this effort, as this is a pretty fundamental change in the NetarchiveSystem. We hope to have initial experimental H3 support ready for a November? Devel release, with a Prod release at the end of the year perhaps ??

Best
Mikis


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.sbforge.org/pipermail/netarchivesuite-devel/attachments/20140507/d59ee5f1/attachment-0001.html>