Nexus Indexer

Skip to end of metadata
Go to start of metadata

Maven Integration for Eclipse is extensively using local index for Maven repositories. There are several repository types supported:

  • remote Maven repositories, such as Central repository at http://repo1.maven.org/maven2/
  • local Maven repository
  • Maven projects available in the Eclipse workspace

All those indexes are managed by Nexus indexer component, which is also used by Maven integration for NetBeans, Sonatype repository manager and Nexus indexer command line tool (CLI).

Nexus indexer component provides an API to index Maven repository, merge and download index updates. It also provides an API to search through registered indexes using various search criteria, including:

Nexus indexer CLI

Nexus indexer CLI can be used to index arbitrary Maven 2 repository and it is packaged as an executable nexus-indexer-XXX-cli.jar. It can be also downloaded from the Nexus project web site.

To get list of available options you can run it without parameters:

>java -jar nexus-indexer-1.0.0-SNAPSHOT-cli.jar
Options:
 -r,--repository    Required path to the Maven repository to be indexed
 -i,--index         Required path to the folder where baseline the
                    index is stored.
 -n,--name          Repository name.
 -t,--type          Indexer type (default, min or full).
 -d,--target        Optional folder name where to publish files
 -v,--version       Display version information

To index repository you need to specify repository name/id and path to repository on local file system (you can't index repository over http, ftp or scp). Additional parameters can be specified to set name of the directory where index will be created as well as index type.

When index folder contains previously created index, the tool will use it as a base line and will generate chunks for the incremental updates.

The indexer types could be one of default, min or full. You can also specify list of coma-separated custom index creators. An index creator should be a regular Plexus component, where its hint property would be the index type used in the command line.

Here is a command we use to index Central Maven repository:

>java -jar nexus-indexer-1.0.0-SNAPSHOT-cli.jar -t full -i central -n central -r {path to repository on a local fs}

After scanning of Maven repository it will create index folder and several files:

nexus-maven-repository-index.properties
nexus-maven-repository-index.zip
nexus-maven-repository-index.gz
nexus-maven-repository-index.yyyyMMdd.gz

A .zip and .gz files contains legacy and the new transfer formats for repository index. Files with the yyyyMMdd are the incremental chunks containing index changes from the yyyyMMdd date up to now.The .properties file contains repository id, last update timestamp and list of published incremental updates:

#Mon Mar 10 14:19:48 CDT 2008
nexus.index.time=20080310141948.414 -0500
nexus.index.update-1=20081129000000.000 -0600
nexus.index.update-0=20081130000000.000 -0600
nexus.index.id=central

Those files need to be copied to /.index folder at repository root. See for example http://repo1.maven.org/maven2/.index/.

Nexus Repository Manager

Sonatype Nexus automatically indexes proxyed and hosted Maven repositories. See the project web site for more details.

Pre-configured indexes

Index configuration can be declared using org.maven.ide.eclipse.indexes extension point. It allows to specify repository url or even prepackaged index bundle and install it as a custom plugin dependent on the org.maven.ide.eclipse plugin. Here is how extension point declared for Maven central repository.

<extension point="org.maven.ide.eclipse.indexes">
    <index indexId="central"
           archive="central.zip"
           isShort="false"
           repositoryUrl="http://repo1.maven.org/maven2/"
           updateUrl="http://www.sonatype.com/nexus/"/>
  </extension>

Note that central.zip is the same archive as nexus-maven-repository-index.zip produced by the Nexus indexer CLI.

Nexus Indexer API Example

The following code snippet shows how to obtain NexusIndexer and IndexUpdater components from MavenEmbedder.

This should be done once and instance reused while application is running.
MavenEmbedder embedder;

// get NexusIndexer component from Plexus
PlexusContainer plexus = embedder.getPlexusContainer();
NexusIndexer indexer = (NexusIndexer) plexus.lookup(NexusIndexer.class);
IndexUpdater updater = (IndexUpdater) plexus.lookup(IndexUpdater.class);

// add indexing context (stateful), should be done once for lifetime
indexer.addIndexingContext(
  indexId,         // index id (usually the same as repository id)
  repositoryId,    // repository id
  directory,       // Lucene directory where index is stored
  repositoryDir,   // local repository dir or null for remote repo
  repositoryUrl,   // repository url, used by index updater
  indexUpdateUrl,  // index update url or null if derived from repositoryUrl
  false, false);

The following snippet shows how to update/download remote index:

IndexingContext context = indexer.getIndexingContexts().get(indexId);

Settings settings = embedder.getSettings();
Proxy proxy = settings.getActiveProxy();
ProxyInfo proxyInfo = null;
if(proxy != null) {
  proxyInfo = new ProxyInfo();
  proxyInfo.setHost(proxy.getHost());
  proxyInfo.setPort(proxy.getPort());
  proxyInfo.setNonProxyHosts(proxy.getNonProxyHosts());
  proxyInfo.setUserName(proxy.getUsername());
  proxyInfo.setPassword(proxy.getPassword());
}

Date indexTime = updater.fetchAndUpdateIndex(context, transferListener, proxyInfo);
...

The following snippet shows how to run search query:

// run search query
BooleanQuery q = new BooleanQuery();
q.add(indexer.constructQuery(ArtifactInfo.GROUP_ID, term), Occur.SHOULD);
q.add(indexer.constructQuery(ArtifactInfo.ARTIFACT_ID, term), Occur.SHOULD);
q.add(new PrefixQuery(new Term(ArtifactInfo.MD5, term)), Occur.SHOULD);
q.add(new PrefixQuery(new Term(ArtifactInfo.SHA1, term)), Occur.SHOULD);

FlatSearchRequest request = new FlatSearchRequest(q);
FlatSearchResponse response = indexer.searchFlat(request);
...

Labels

nexus nexus Delete
m2eclipse m2eclipse Delete
maven maven Delete
eclipse eclipse Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Mar 12, 2008

    linux_china@hotmail.com says:

    I have a questions. Can you consider to build dependency into the index? Depende...

    I have a questions. Can you consider to build dependency into the index? Dependency information is very useful for artifact relation.

  2. Mar 12, 2008

    Eugene Kuleshov says:

    Jacky, it shouldn't be too difficult to include additional info into the index, ...

    Jacky, it shouldn't be too difficult to include additional info into the index, but can you please elaborate little bit more on the use case or better submit an enhancement request to the project issue tracker. Thanks.

  3. Mar 13, 2008

    Milos Kleint says:

    +1 on jacky's suggestion. I've tried to extend the index for the netbeans integr...

    +1 on jacky's suggestion.
    I've tried to extend the index for the netbeans integration, so it only works for local repository. (sort of)

    It's useful to find what other projects use a particular library in order to check their sources for example usage or just to guess the popularity of a library. In a enterprise environment it's yet more useful to find all clients of your api for example if you do code optimization or break the api and need to fix the clients..

  4. Mar 13, 2008

    Justin Edelson says:

    I've tried running java -jar nexus-indexer-1.0.0-SNAPSHOT-cli.jar -o -z -t full...

    I've tried running

    java -jar nexus-indexer-1.0.0-SNAPSHOT-cli.jar -o -z -t full -i inhouse -n inhouse -r /foo/bar
    

    several times now and I get a directory containing four files:

    • _2.cfs
    • segments_3
    • segments.gen
    • timestamp

    Any ideas?

  5. Mar 13, 2008

    Eugene Kuleshov says:

    Justin, are you saying it didn't created nexus-maven-repository-index.properties...

    Justin, are you saying it didn't created nexus-maven-repository-index.properties and nexus-maven-repository-index.zip files in the directory you are running it from? Please check that you have writing permissions for that dir and add -X -e options to see if there was any errors.

  6. Mar 14, 2008

    Justin Edelson says:

    Ah. I see my error. The nexus-maven-repository-index.properties and nexus-maven-...

    Ah. I see my error. The nexus-maven-repository-index.properties and nexus-maven-repository-index.zip files are created in the working directory. I was looking in the index directory.

    Separate question - is there any way to use the index client in a standalone application?

  7. Mar 18, 2008

    linux_china@hotmail.com says:

    Hi  Eugene, dependency is very important to management classpath, so I can ...

    Hi  Eugene, dependency is very important to management classpath, so I can build the classpath for the project even without pom file.  Another consider is about reference. I want to know how many projects use special artifact. There are broken dependency relation between projects, and we can use depency index to find broken links.

    I have another question about code completion for plugin configuration parameters, I want to implement this feature in IDE, and I can not find plugin parameters information. Maybe I ask you to build these information into the index.   I have no idea how to implement it now.

  8. Mar 31, 2008

    ghandim says:

    Hi Eugene, very good feature. Also, i have one problem. If i try to update in Ec...

    Hi Eugene,
    very good feature. Also, i have one problem. If i try to update in Eclipse an by my own generated index it always fails, because a file timestamp cannot overwritten.

    Best regards, Michael

  9. Mar 31, 2008

    Eugene Kuleshov says:

    Michael, please open a jira issue in m2eclipse project and provide the stack tra...

    Michael, please open a jira issue in m2eclipse project and provide the stack trace.

  10. Apr 02, 2008

    baljit.singh@chordiant.com says:

    How can I define a new index in Eclipse Europa, like  Maven centr...

    How can I define a new index in Eclipse Europa, like  Maven central repository.

    What I did is installed maven plugin for Eclipse Europa. In Eclipse, in Maven Indexes View, right click and selected Add Index, provided Respository URL and ID. After restarting Eclipse the entry is missing from Maven Indexed View.

    Is this a bug or I am doing something wrong?

    Thanks

    Baljit 

  11. Apr 02, 2008

    Eugene Kuleshov says:

    Baljit, generally it is better to address such questions to m2eclipse users mail...

    Baljit, generally it is better to address such questions to m2eclipse users mailing list.

    Anyways, to be browseable, the repository you added to Maven Indexes view should have nexus indexes deployed. If you own that repository you can use Nexus Indexer CLI to index it and publish indexes, otherwise you need to ask repository owners to index it as described on the wiki page above.

  12. Jul 29, 2008

    liliya pesotska says:

    Hi, is there any documentation about how to query the nexus index programmatical...

    Hi, is there any documentation about how to query the nexus index programmatically (tutorial, examples etc.)?

  13. Jul 29, 2008

    Eugene Kuleshov says:

    Liliya, please see example above. You may also look at m2eclipse code, e.g. Nexu...

    Liliya, please see example above. You may also look at m2eclipse code, e.g. NexusIndexManager class.

  14. Aug 05, 2008

    memter says:

    Hi!   Could we use Nexus Indexer for index snapshot repository? When I...

    Hi!

     
    Could we use Nexus Indexer for index snapshot repository? When I index snapshot repository, the created index doesn't have any artifacts that's ends with -SNAPSHOT. Only artifacts with packaging 'pom' were indexed in snapshot repository.

  15. Aug 05, 2008

    Tamás Cservenák says:

    Memter, i believe you are using some old version of Nexus Indexer. Nexus indexe...

    Memter,

    i believe you are using some old version of Nexus Indexer. Nexus indexer really had a nasty bug about leaving out snapshots, but it was reported long ago, and fixed. See:

    http://issues.sonatype.org/browse/NEXUS-13

    If you are using Nexus Indexer CLI, you can get the newest release from here (nexus-indexer-1.0-beta-4.3-cli.jar):

    http://repository.sonatype.org/content/repositories/releases/org/sonatype/nexus/nexus-indexer/1.0-beta-4.3/

    And latest CI build (not recommended for production sites) from here (pick the latest build):

    http://repository.sonatype.org/content/repositories/snapshots/org/sonatype/nexus/nexus-indexer/1.0-beta-5-SNAPSHOT/

    Hope helps,

    Tamas

  16. Aug 05, 2008

    memter says:

    Yes, You were right. I've updated Nexus Indexer and now it works great Thanks T...

    Yes, You were right. I've updated Nexus Indexer and now it works great Thanks Tamas Cservenak!

  17. Sep 04, 2008

    locutusut says:

    Hey I ran this tool using the following command against my snapshot repository: ...

    Hey I ran this tool using the following command against my snapshot repository:

    $ java -jar nexus-indexer-1.0.0-cli.jar -o -z -t full -i snapshots -n snapshots -r ./snapshots
    Scanning started
    Total files scanned: 4
    Total time: 0 sec
    Final memory: 1M/31M

    It created the zip/properties files, but it completely wiped out the contents of my SNAPSHOT repo...why would it do that?? Thank God I had a backup of the repository. I was expecting it to create the index not blow away my repository!

    Please advise...

    Joe

  18. Sep 04, 2008

    Eugene Kuleshov says:

    Joe, you are NOT supposed to set index folder to the same place as your Maven re...

    Joe, you are NOT supposed to set index folder to the same place as your Maven repository. Because you didn't specify "incremental" flag, index folder is deleted before indexing. I suppose indexer tool could be more fool proof, but for now please use separate temporary folder for the index and move zip and property files into <repo>/.index/ after indexing.

  19. Sep 08, 2008

    locutusut says:

    Thanks Eugene...that worked, however, I don't see mention above of an "increment...

    Thanks Eugene...that worked, however, I don't see mention above of an "incremental" flag.  Do you mean omitting the -o flag?

    Also, I assume you can set this up on a cron job to incrementally update the index?  It sounds like there is no reason to maintain the "index" folder after the zip is deleted so that could be removed each time.

  20. Sep 08, 2008

    Eugene Kuleshov says:

    Joe, sorry about confusion, I was referring to -u (create update) option. It is ...

    Joe, sorry about confusion, I was referring to -u (create update) option. It is significantly faster then full index, but we know at least one gap in the current implementation. For example, it does not remove entries from the index if artifacts been deleted from repository. We plan to expand this mode in the future and cover corner cases to speed up updating indexes for the large repositories.